Image processing device and method

ABSTRACT

The present invention relates to image processing device and method which can suppress block noise. A filter strength determination unit determines the filter strength for every four lines under the control of a control unit. In other words, if a block boundary determination unit has determined to perform filtering, the filter strength determination unit determines at which strength of the strong filter or the weak filter the filtering process is performed, and outputs the determination result to a filter calculation unit. On this occasion, the filter strength determination unit uses the Bs value in the determination formula of the strong filter. For example, in the case of using the Bs value, the threshold of the determination is set in accordance with a linear function of the Bs value from the control unit. The present disclosure can be applied to, for example, an image processing device.

TECHNICAL FIELD

The present disclosure relates to image processing device and method,and particularly to image processing device and method that can suppressblock noise.

BACKGROUND ART

In recent years, a device has become popular that handles imageinformation digitally and for the purpose of highly efficientlytransmitting and accumulating pieces of information, compresses andencodes an image by employing an encoding method that compresses theimage through the motion compensation and orthogonal transform such asdiscrete cosine transform by using the redundancy unique to the imageinformation. This encoding method includes, for example, MPEG (MovingPicture Experts Group) and H.264 and MPEG-4 Part 10 (Advanced VideoCoding, hereinafter referred to as H.264/AVC).

Recently, for the purpose of further improving the encoding efficiencyover H.264/AVC, JCTVC (Joint Collaboration Team-Video Coding) as theITU-T and ISO/IEC joint standardization group has advanced thestandardization of the encoding method called HEVC (High EfficiencyVideo Coding) (for example, see Non-Patent Document 1).

In the current draft of HEVC, a deblocking filter is employed as one ofin-loop filters. As the deblocking filters, a strong filter and a weakfilter are given; which one is employed is determined by using a filterdetermination formula for the block boundary.

In other words, whether to apply the strong filter or not is determinedusing the determination formula of the strong filter and for the blockboundary where it has been determined that the strong filter is notapplied, whether to apply the weak filter or not is determined using thedetermination formula of the weak filter.

CITATION LIST Non-Patent Document

-   Non-Patent Document 1: Benjamin Bross, Woo-Jin Han, Jens-Rainer Ohm,    Gary J. Sullivan, Thomas Wiegand, “High efficiency video coding    (HEVC) text specification draft 8”, JCTVC-J1003_d7, 2012 Jul. 28

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

However, if the determination formula of the strong filter in the HEVCdraft is still used, the weak filter is often applied in the portionwhere the block noise is remarkable, in which case the block noise mayremain.

The present disclosure has been made in view of the above and cansuppress the block noise.

Solutions to Problems

An image processing device according to a first aspect of the presentdisclosure includes: a filter determination unit that determines whetherto apply a strong filter by using a strength parameter representingstrength of a deblocking filter for a block boundary, which is aboundary between a block of an image locally decoded when an image isencoded, and an adjacent block adjacent to the block; a filter unit thatapplies the strong filter for the block boundary if the filterdetermination unit has determined to apply the strong filter; and anencoding unit that encodes the image in units having a layer structureby using the image to which the strong filter has been applied by thefilter unit.

The filter determination unit can determine whether to apply the strongfilter by using a strength parameter representing the strength of thedeblocking filter for the block boundary and a determination parameterrepresenting a threshold used in the determination of the deblockingfilter.

Whether to apply the strong filter can be determined by using a valueobtained by multiplying the strength parameter by the determinationparameter.

The filter determination unit can determine whether to apply the strongfilter by using a function in which the strength parameter or thedetermination parameter is a variable.

The filter determination unit can determine whether to apply the strongfilter by using a linear function of the strength parameter and thedetermination parameter.

The strength parameter is larger when a prediction mode is intraprediction than when the prediction mode is inter prediction.

The strength parameter is a value of Bs (Boundary Filtering Strength).

The determination parameter is beta used in the determination of whetherto apply the deblocking filter and the determination of the strengthselection of the deblocking filter.

In an image processing method according to a first aspect of the presentdisclosure, an image processing device performs the steps of:determining whether to apply a strong filter by using a strengthparameter representing strength of a deblocking filter for a blockboundary, which is a boundary between a block of an image locallydecoded when an image is encoded, and an adjacent block adjacent to theblock; applying the strong filter for the block boundary if it has beendetermined to apply the strong filter; and encoding the image in unitshaving a layer structure by using the image to which the strong filterhas been applied.

In the first aspect of the present disclosure, it is determined whetherto apply a strong filter by using a strength parameter representingstrength for a block boundary, which is a boundary between a block of animage locally decoded when an image is encoded, and an adjacent blockadjacent to the block. If it has been determined to apply the strongfilter, the strong filter is applied for the block boundary, and theimage is encoded in units having a layer structure by using the image towhich the strong filter has been applied.

The image processing device may be an independent device or an internalblock included in one image encoding device or image decoding device.

Effects of the Invention

According to the first aspect of the present disclosure, the image canbe encoded, and particularly the block noise can be suppressed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a main structure example of animage encoding device.

FIG. 2 is a flowchart for describing an example of the flow of anencoding process.

FIG. 3 is a block diagram illustrating a main structure example of animage decoding device.

FIG. 4 is a flowchart for describing an example of the flow of adecoding process.

FIG. 5 is a diagram for describing a determination formula of a strongfilter.

FIG. 6 is a diagram illustrating the relation between a quantizationparameter and a value of the right side of Formula (4) when f1(Bs)=Bs+2.

FIG. 7 is a diagram illustrating an example of pixel values of a block Pand an adjacent block Q of FIG. 5 in the case of FIG. 6.

FIG. 8 is a diagram illustrating the relation between the quantizationparameter and a value of the right side of Formula (5) when f2(Bs)=Bs+2.

FIG. 9 is a diagram illustrating an example of pixel values of the blockP and the adjacent block Q of FIG. 5 in the case of FIG. 8.

FIG. 10 is a diagram illustrating the relation between the quantizationparameter and a value of the right side of Formula (6) when f3(Bs)=Bs+2.

FIG. 11 is a diagram illustrating an example of pixel values of theblock P and the adjacent block Q of FIG. 5 in the case of FIG. 10.

FIG. 12 is a diagram for describing a specific example.

FIG. 13 is a block diagram illustrating a structure example of adeblocking filter to which the present technique has been applied.

FIG. 14 is a flowchart for describing a deblocking filtering process.

FIG. 15 is a flowchart for describing a filtering process for aluminance boundary.

FIG. 16 is a diagram illustrating an example of a multi-viewpoint imageencoding method.

FIG. 17 is a diagram illustrating a main structure example of amulti-viewpoint image encoding device to which the present technique hasbeen applied.

FIG. 18 is a diagram illustrating a main structure example of amulti-viewpoint image decoding device to which the present technique hasbeen applied.

FIG. 19 is a diagram illustrating an example of the multi-viewpointimage encoding method.

FIG. 20 is a diagram illustrating a main structure example of themulti-viewpoint image encoding device to which the present technique hasbeen applied.

FIG. 21 is a diagram illustrating a main structure example of themulti-viewpoint image decoding device to which the present technique hasbeen applied.

FIG. 22 is a block diagram illustrating a main structure example of acomputer.

FIG. 23 is a block diagram illustrating an example of a schematicstructure of a television device.

FIG. 24 is a block diagram illustrating an example of a schematicstructure of a cellular phone.

FIG. 25 is a block diagram illustrating an example of a schematicstructure of a recording/reproducing device.

FIG. 26 is a block diagram illustrating an example of a schematicstructure of a photographing device.

FIG. 27 is a block diagram illustrating an example of a schematicstructure of a video set.

FIG. 28 is a block diagram illustrating an example of a schematicstructure of a video processor.

FIG. 29 is a block diagram illustrating another example of a schematicstructure of a video processor.

MODE FOR CARRYING OUT THE INVENTION

A mode for carrying out the present disclosure (hereinafter referred toas an embodiment) is hereinafter described in the following order:

1. Summary of device and operation

2. First embodiment (image encoding device, image decoding device)

3. Second embodiment (multi-viewpoint image encoding/decoding)

4. Third embodiment (layer image encoding/decoding)

5. Fourth embodiment (computer)

6. Application example

7. Fifth embodiment (set, unit, module, processor)

1. Summary of Device and Operation

[Structure Example of Image Encoding Device]

FIG. 1 illustrates a structure of one embodiment of an image encodingdevice as an image processing device to which the present disclosure hasbeen applied.

An image encoding device 11 illustrated in FIG. 1 encodes the image datathrough a prediction process. As the encoding method, for example, theHEVC (High Efficiency Video Coding) method or the like is used.

Note that the coding unit (CU (Coding Unit)) is defined in the HEVCmethod. CU is also referred to as Coding Tree Block (CTB) and is thepartial region of the image in units of picture that plays a rolesimilar to the macroblock in the H.264/AVC method. While the latter isfixed to the size of 16×16 pixels, the size of the former is not fixedand will be specified in the image compression information in eachsequence.

For example, in the sequence parameter set (SPS (Sequence ParameterSet)) included in the encoded data to be the output, the maximum size ofCU (LCU (Largest Coding Unit)) and the minimum size of CU (SCU (SmallestCoding Unit)) are defined.

In each LCU, by setting to split−flag=1 in the range that the size doesnot become less than the size of SCU, the unit can be divided into thesmaller CUs. When the split_flag has a value of “1”, the CU with a sizeof 2N×2N is divided into CUs with a size of N×N in a one-lower layer.

Moreover, the CU is divided into prediction units (Prediction Units(PUs)), each region serving as a region to be subjected to the intraprediction or inter prediction (partial region of the image in units ofpicture), and into transform units (Transform Units (TUs)), each regionserving as a region to be subjected to the orthogonal transform (partialregion of the image in units of picture). At present, in the HEVCmethod, in addition to the 4×4 and 8×8 orthogonal transforms, 16×16 and32×32 orthogonal transforms can be used.

The image encoding device 11 of FIG. 1 includes an A/D (Analog/Digital)conversion unit 21, a screen rearrangement buffer 22, a calculation unit23, an orthogonal transform unit 24, a quantization unit 25, a losslessencoding unit 26, and an accumulation buffer 27. The image encodingdevice 11 includes an inverse quantization unit 28, an inverseorthogonal transform unit 29, a calculation unit 30, a deblocking filter31 a, a frame memory 32, a selection unit 33, an intra prediction unit34, a motion prediction/compensation unit 35, a predicted imageselection unit 36, and a rate control unit 37.

Moreover, the image encoding device 11 includes an adaptive offsetfilter 41 and an adaptive loop filter 42 between the deblocking filter31 a and the frame memory 32.

The A/D conversion unit 21 performs the A/D conversion of the inputimage data and outputs the data to the screen rearrangement buffer 22and stores the data therein.

The screen rearrangement buffer 22 rearranges the images, whose frameshave been arranged in the order of display as stored, in the order ofthe encoding in accordance with GOP (Group of Picture). The screenrearrangement buffer 22 supplies the images whose frames have beenrearranged, to the calculation unit 23. The screen rearrangement buffer22 also supplies the images whose frames have been rearranged, to theintra prediction unit 34 and the motion prediction/compensation unit 35.

The calculation unit 23 subtracts the predicted image supplied from theintra prediction unit 34 or the motion prediction/compensation unit 35through the predicted image selection unit 36 from the image read outfrom the screen rearrangement buffer 22, and outputs the differentialinformation to the orthogonal transform unit 24.

For example, in the case of the image for which the intra-encoding isperformed, the calculation unit 23 subtracts the predicted imagesupplied from the intra prediction unit 34 from the image read out fromthe screen rearrangement buffer 22. Moreover, in the case of the imagefor which the inter-encoding is performed, the calculation unit 23subtracts the predicted image supplied from the motionprediction/compensation unit 35 from the image read out from the screenrearrangement buffer 22.

The orthogonal transform unit 24 performs the orthogonal transform suchas the discrete cosine transform or Karhunen-Loeve transform on thedifferential information supplied from the calculation unit 23. Theorthogonal transform unit 24 supplies the transform coefficient to thequantization unit 25.

The quantization unit 25 quantizes the transform coefficient output fromthe orthogonal transform unit 24. The quantization unit 25 supplies thequantized transform coefficient to the lossless encoding unit 26.

The lossless encoding unit 26 performs the lossless encoding such as thevariable-length encoding or arithmetic encoding on the quantizedtransform coefficient.

The lossless encoding unit 26 acquires the parameter such as theinformation representing the intra prediction mode from the intraprediction unit 34, and acquires the parameter such as the informationrepresenting the inter prediction mode or the motion vector informationfrom the motion prediction/compensation unit 35.

The lossless encoding unit 26 encodes the quantized transformcoefficient and encodes the acquired parameters (syntax elements) andmakes these a part of the header information of the encoded data(multiplexing). The lossless encoding unit 26 supplies the encoded datato the accumulation buffer 27 and accumulates the data therein.

For example, in the lossless encoding unit 26, the lossless encodingprocess such as the variable-length encoding or the arithmetic encodingis conducted. As the variable-length encoding, for example, CAVLC(Context-Adaptive Variable Length Coding) is given. As the arithmeticencoding, for example, CABAC (Context-Adaptive Binary Arithmetic Coding)is given.

The accumulation buffer 27 temporarily holds the encoded stream (data)supplied from the lossless encoding unit 26. The accumulation buffer 27outputs the encoded data as the encoded image to, for example, atransmission path or a recording device in the later stage, which is notshown, at a predetermined timing. In other words, the accumulationbuffer 27 also serves as a transmission unit that transmits the encodedstream.

The transform coefficient quantized in the quantization unit 25 is alsosupplied to the inverse quantization unit 28. The inverse quantizationunit 28 inversely quantizes the quantized transform coefficient by amethod corresponding to the quantization by the quantization unit 25.The inverse quantization unit 28 supplies the obtained transformcoefficient to the inverse orthogonal transform unit 29.

The inverse orthogonal transform unit 29 performs the inverse orthogonaltransform on the supplied transform coefficient by a methodcorresponding to the orthogonal transform process by the orthogonaltransform unit 24. The output that has been subjected to the inverseorthogonal transform (recovered differential information) is supplied tothe calculation unit 30.

The calculation unit 30 adds the predicted image supplied from the intraprediction unit 34 or the motion prediction/compensation unit 35 throughthe predicted image selection unit 36 to the inverse orthogonaltransform result supplied from the inverse orthogonal transform unit 29,i.e., the recovered differential information, thereby providing thelocally decoded image (decoded image).

For example, if the differential information corresponds to the imagefor which the intra encoding is performed, the calculation unit 30 addsthe predicted image supplied from the intra prediction unit 34 to thedifferential information. For example, if the differential informationcorresponds to the image for which the inter encoding is performed, thecalculation unit 30 adds the predicted image supplied from the motionprediction/compensation unit 35 to the differential information.

The decoded image, which is the result of addition, is supplied to thedeblocking filter 31 a and the frame memory 32.

The deblocking filter 31 a suppresses the block distortion of thedecoded image by performing the deblocking filtering process asappropriate. The deblocking filter 31 a has the strong (strong) filterand weak (weak) filter, and which filter is to be used is determined byusing determination formulae. The determination formulae of the strongfilter include three determination formulae, and each employs a value ofBs (Boundary Filtering Strength) in addition to the parameter beta or tcobtained based on the quantization parameter QP. The parameters beta andtc are the thresholds used in the determination related to the filter,and are the determination parameters used in determining whether toapply the deblocking filter and determining the strength selection. Theparameter tc is the parameter also used for the filter itself. The valueof Bs is the strength parameter representing the strength for the blockboundary.

The deblocking filter 31 a can be disabled by the user instruction, andthe ON/OFF information representing whether to apply the deblockingfilter or not is input to the deblocking filter 31 a by the user throughthe operation of the operation unit, which is not shown. Each offset ofthe aforementioned parameters beta and tc is 0 by default, and if thevalue is not 0, the value is input to the deblocking filter 31 a by theuser operation through the operation unit, which is not shown. TheON/OFF information (also referred to as DisableDeblockingFilter flaginformation) of the deblocking filter and each offset of the parametersbeta and tc are encoded in the lossless encoding unit 26 as theparameters of the deblocking filter, and transmitted to an imagedecoding device 51 of FIG. 3 as described below.

The deblocking filter 31 a performs the deblocking filtering processusing the aforementioned ON/OFF information, the offset, the Bs value,the parameters beta and tc, and the like for the image from thecalculation unit 30. The deblocking filter 31 a supplies the filteringprocess result to the adaptive offset filter 41. Note that the detailedstructure of the deblocking filter 31 a is described below withreference to FIG. 13.

The adaptive offset filter 41 performs the offset filter (SAO: Sampleadaptive offset) process especially for suppressing ringing for theimage after the filtering with the deblocking filter 31 a.

The types of the offset filters include two type of band offset, sixtypes of edge offset, and non-offset, which are nine types in total. Theadaptive offset filter 41 performs the filtering process for the imageafter the filtering with the deblocking filter 31 a using the offsetvalue for each region (such as LOU), for which the type of offset filteris decided for each LCU. The adaptive offset filter 41 supplies theimage after the filtering process to the adaptive loop filter 42.

In the image encoding device 11, the offset value for each LCU iscalculated by the adaptive offset filter 41 and used. The calculatedoffset value for each LCU is encoded in the lossless encoding unit 26 asthe adaptive offset parameter, and transmitted to the image decodingdevice 51 of FIG. 3, which is described below.

The adaptive loop filter 42 performs the adaptive loop filter (ALF:AdaptiveLoop Filter) process on the image after the filtering with theadaptive offset filter 41 in units of process (such as LCU) using afilter coefficient. As the adaptive loop filter 42, for example, atwo-dimensional Wiener Filter (Wiener Filter) is used. Needless to say,a filter other than the Wiener Filter may be used. The adaptive loopfilter 42 supplies the filtering process result to the frame memory 32.

Although not illustrated in the example of FIG. 1, the filtercoefficient in the image encoding device 11 is calculated by theadaptive loop filter 42 in units of process so that the difference fromthe original image from the screen rearrangement buffer 12 is minimized,and then used. The calculated filter coefficient is encoded in thelossless encoding unit 26 as the adaptive loop filter parameter andtransmitted to the image decoding device 51 of FIG. 3, which isdescribed below.

The frame memory 32 outputs the accumulated reference images at apredetermined timing to the intra prediction unit 34 or the motionprediction/compensation unit 35 through the selection unit 33.

For example, in the case of the image for which the intra encoding isperformed, the frame memory 32 supplies the reference image to the intraprediction unit 34 through the selection unit 33. For example, in thecase where the inter encoding is performed, the frame memory 32 suppliesthe reference image to the motion prediction/compensation unit 35through the selection unit 33.

If the reference image supplied from the frame memory 32 is the imagefor which the intra encoding is performed, the selection unit 33supplies the reference image to the intra prediction unit 34. Moreover,if the reference image supplied from the frame memory 32 is the imagefor which the inter encoding is performed, the selection unit 33supplies the reference image to the motion prediction/compensation unit35.

The intra prediction unit 34 performs the intra prediction (in-screenprediction) for generating the predicted image using the pixel value inthe screen. The intra prediction unit 34 performs the intra predictionin a plurality of modes (intra prediction mode).

The intra prediction unit 34 generates the predicted image in all theintra prediction modes, evaluates each predicted image, and then selectsthe optimum mode. Upon the selection of the optimum intra predictionmode, the intra prediction unit 34 supplies the predicted imagegenerated in that optimum mode to the calculation unit 23 or thecalculation unit 30 through the predicted image selection unit 36.

As described above, the intra prediction unit 34 supplies the intraprediction mode information representing the employed intra predictionmode to the lossless encoding unit 26 as appropriate.

In regard to the image for which the inter encoding is performed, themotion prediction/compensation unit 35 performs the motion predictionusing the input image supplied from the screen rearrangement buffer 22and the reference image supplied from the frame memory 32 through theselection unit 33. The motion prediction/compensation unit 35 generatesthe predicted image (inter predicted image information) through themotion compensation process according to the motion vector detected bythe motion prediction.

The motion prediction/compensation unit 35 generates the predicted imageby performing the inter prediction process in all the inter predictionmode candidates. The motion prediction/compensation unit 35 supplies thegenerated predicted image to the calculation unit 23 or the calculationunit 30 through the predicted image selection unit 36.

The motion prediction/compensation unit 35 supplies the parameters suchas the inter prediction mode information representing the employed interprediction mode and the motion vector information representing thecalculated motion vector to the lossless encoding unit 26.

In the case of the image for which the intra encoding is performed, thepredicted image selection unit 36 supplies the output of the intraprediction unit 34 to the calculation unit 23 or the calculation unit30; in the case of the image for which the inter encoding is performed,the predicted image selection unit 36 supplies the output of the motionprediction/compensation unit 35 to the calculation unit 23 or thecalculation unit 30.

The rate control unit 37 controls the rate of the quantization operationof the quantization unit 25 based on the compressed image accumulated inthe accumulation buffer 27 so that the overflow or the underflow doesnot occur.

[Operation of Image Encoding Device]

With reference to FIG. 2, the flow of the encoding process to beexecuted by the aforementioned image encoding device 11 is described.

In step S11, the A/D conversion unit 21 performs the A/D conversion ofthe input images. In step S12, the screen rearrangement buffer 22 storesthe images after the A/D conversion, and rearranges the pictures, whichhave been arranged in the order of display, in the order of encoding.

If the image to be processed, which is supplied from the screenrearrangement buffer 22, is the image of the block for which the intraprocess is performed, the decoded image to be referred to is read out ofthe frame memory 32 and supplied to the intra prediction unit 34 throughthe selection unit 33.

Based on these images, in step S13, the intra prediction unit 34performs the intra prediction on the pixels in the block to be processedin all the intra prediction mode candidate. As the decoded pixel to bereferred to, the pixel for which the filtering with the deblockingfilter 31 a is not performed is used.

Through this process, the intra prediction is performed in all the intraprediction mode candidates, and the cost function value is calculatedfor all the intra prediction mode candidates. Based on the calculatedcost function value, the optimum intra prediction mode is selected andthe predicted image generated by the intra prediction in the optimumintra prediction mode and the cost function value are supplied to thepredicted image selection unit 36.

If the image to be processed, which is supplied from the screenrearrangement buffer 22, is the image for which the inter process isperformed, the image to be referred to is read out of the frame memory32 and supplied to the motion prediction/compensation unit 35 throughthe selection unit 33. Based on these images, in step S14, the motionprediction/compensation unit 35 performs the motionprediction/compensation process.

Through this process, the motion prediction process is performed in allthe inter prediction mode candidates, the cost function value iscalculated for all the inter prediction mode candidates, and the optimuminter prediction mode is decided based on the calculated cost functionvalues. Then, the predicted image generated by the optimum interprediction mode and the cost function value thereof are supplied to thepredicted image selection unit 36.

In step S15, the predicted image selection unit 36 decides one of theoptimum intra prediction mode and the optimum inter prediction mode asthe optimum prediction mode based on each cost function value outputfrom the intra prediction unit 34 and the motion prediction/compensationunit 35. Then, the predicted image selection unit 36 selects thepredicted image of the decided optimum prediction mode and supplies theimage to the calculation units 23 and 30. This predicted image is usedin the calculation in steps S16 and S21 as described below.

The selection information for this predicted image is supplied to theintra prediction unit 34 or the motion prediction/compensation unit 35.If the predicted image in the optimum intra prediction mode has beenselected, the intra prediction unit 34 supplies the informationrepresenting the optimum intra prediction mode (i.e., parameter relatedto the intra prediction) to the lossless encoding unit 26.

If the predicted image in the optimum inter prediction mode has beenselected, the motion prediction/compensation unit 35 outputs theinformation representing the optimum inter prediction mode and theinformation based on the optimum inter prediction mode (i.e., parameterrelated to the motion prediction) to the lossless encoding unit 26. Theinformation based on the optimum inter prediction mode corresponds tomotion vector information, reference frame information, or the like.

In step S16, the calculation unit 23 calculates the difference betweenthe image rearranged in step S12 and the predicted image selected instep S15. In the case of the inter prediction, the predicted image issupplied from the motion prediction/compensation unit 35 to thecalculation unit 23 through the predicted image selection unit 36; inthe case of the intra prediction, the predicted image is supplied fromthe intra prediction unit 34 to the calculation unit 23 through thepredicted image selection unit 36.

The differential data are smaller in data quantity than the originalimage data. Therefore, as compared to the case in which the image isencoded without any process, the data quantity can be compressed.

In step S17, the orthogonal transform unit 24 performs the orthogonaltransform of the differential information supplied from the calculationunit 23. Specifically, the orthogonal transform such as the discretecosine transform or Karhunen-Loeve transform is performed, and thetransform coefficient is output.

In step S18, the quantization unit 25 quantizes the transformcoefficient. In this quantization, the rate is controlled as describedbelow in the process of step S28.

The differential information quantized in this manner is locally decodedas below. In step S19, the inverse quantization unit 28 inverselyquantizes the transform coefficient quantized by the quantization unit25 with the characteristic corresponding to the characteristic of thequantization unit 25. In step S20, the inverse orthogonal transform unit29 performs the inverse orthogonal transform of the transformcoefficient, which has been inversely quantized by the inversequantization unit 28, with the characteristic corresponding to thecharacteristic of the orthogonal transform unit 24.

In step S21, the calculation unit 30 adds the predicted image inputthrough the predicted image selection unit 36 to the locally decodeddifferential information, thereby generating the locally decoded (i.e.,local-decoded) image (image corresponding to the input to thecalculation unit 23).

In step S22, the deblocking filter 31 a performs the deblockingfiltering process on the image output from the calculation unit 30. Inthe deblocking filtering process, which is described in detail withreference to FIG. 14, the Bs value is used for the determination of thestrong filter. The image after the filtering with the deblocking filter31 a is output to the adaptive offset filter 41.

The ON/OFF information used in the deblocking filter 31 a, which isinput by the user operation through the operation unit or the like, andeach offset of the parameters beta and tc are supplied to the losslessencoding unit 26 as the parameters of the deblocking filter.

In step S23, the adaptive offset filter 41 performs the adaptive offsetfiltering process. Through this process, the filtering process isperformed on the image after the filtering with the deblocking filter 31a using the offset value for each LCU for which the type of offsetfilter is decided for each LCU. The image after the filtering issupplied to the adaptive loop filter 42.

The decided offset value for each LCU is supplied to the losslessencoding unit 26 as the adaptive offset parameter.

In step S24, the adaptive loop filter 42 performs the adaptive loopfiltering process on the image after the filtering with the adaptiveoffset filter 41. For example, the image after the filtering with theadaptive offset filter 41 is subjected to the filtering process in unitsof process using the filter coefficient, and the filtering processresult is supplied to the frame memory 32.

The frame memory 32 stores the filtered image in step S25. To the framememory 32, the images not filtered by the deblocking filter 31, theadaptive offset filter 41, or the adaptive loop filter 42 are alsosupplied from the calculation unit 30 and stored therein.

On the other hand, the transform coefficient quantized in step S18 isalso supplied to the lossless encoding unit 26. In step S26, thelossless encoding unit 26 encodes the quantized transform coefficientoutput from the quantization unit 25, and each supplied parameter. Inother words, the differential image is subjected to the losslessencoding, such as the variable length encoding or the arithmeticencoding, and thus compressed. As each of the parameters to be encoded,the parameter of the deblocking filter, the parameter of the adaptiveoffset filter, the parameter of the adaptive loop filter, thequantization parameter, the motion vector information, the referenceframe information, the prediction mode information, and the like aregiven.

In step S27, the accumulation buffer 27 accumulates the encodeddifferential images (i.e., encoded stream) as the compressed image. Thecompressed images accumulated in the accumulation buffer 27 are read outand transmitted to the decoding side through the transmission path.

In step S28, the rate control unit 37 controls the rate of thequantization operation of the quantization unit 25 based on thecompressed images accumulated in the accumulation buffer 27 so that theoverflow or underflow does not occur.

Upon the end of the process of step S28, the encoding process ends.

[Structure Example of Image Decoding Device]

FIG. 3 illustrates a structure of one embodiment of an image decodingdevice as an image processing device to which the present disclosure hasbeen applied. The image decoding device 51 illustrated in FIG. 3 is adecoding device for the image encoding device 11 of FIG. 1.

The encoded stream (data) encoded by the image encoding device 11 istransmitted to the image decoding device 51 for the image encodingdevice 11 through the predetermined transmission path, and decodedtherein.

As illustrated in FIG. 3, the image decoding device 51 includes anaccumulation buffer 61, a lossless decoding unit 62, an inversequantization unit 63, an inverse orthogonal transform unit 64, acalculation unit 65, a deblocking filter 31 b, a screen rearrangementbuffer 67, and a D/A conversion unit 68. The image decoding device 51includes a frame memory 69, a selection unit 70, an intra predictionunit 71, a motion prediction/compensation unit 72, and a selection unit73.

Moreover, the image decoding device 51 includes an adaptive offsetfilter 81 and an adaptive loop filter 82 between the deblocking filter31 b, and the screen rearrangement buffer 67 and the frame memory 69.

The accumulation buffer 61 also serves as a reception unit that receivesthe transmitted encoded data. The accumulation buffer 61 receives andaccumulates the transmitted pieces of encoded data. The encoded data arethe data obtained by the encoding of the image encoding device 11. Thelossless decoding unit 62 decodes the encoded data, which are read outof the accumulation buffer 61 at a predetermined timing, by a methodcorresponding to the encoding method of the lossless encoding unit 26 ofFIG. 1.

The lossless decoding unit 62 supplies the decoded parameter such as theinformation representing the intra prediction mode to the intraprediction unit 71, and supplies the parameter such as the informationrepresenting the inter prediction mode or the motion vector informationto the motion prediction/compensation unit 72. The lossless decodingunit 62 supplies the decoded parameter of the deblocking filter to thedeblocking filter 31 b, and supplies the decoded adaptive offsetparameter to the adaptive offset filter 81.

The inverse quantization unit 63 inversely quantizes the coefficientdata (quantization coefficient) obtained by the decoding of the losslessdecoding unit 62, by a method corresponding to the quantization methodof the quantization unit 25 of FIG. 1. In other words, the inversequantization unit 63 inversely quantizes the quantization coefficient bya method similar to the method of the inverse quantization unit 28 ofFIG. 1 using the quantization parameter supplied from the image encodingdevice 11.

The inverse quantization unit 63 supplies the inversely quantizedcoefficient data, i.e., the orthogonal transform coefficient to theinverse orthogonal transform unit 64. The inverse orthogonal transformunit 64 performs the inverse orthogonal transform of the orthogonaltransform coefficient by a method corresponding to the orthogonaltransform method of the orthogonal transform unit 24 of FIG. 1, therebyproviding the decoding residual data corresponding to the residual databefore the orthogonal transform in the image encoding device 11.

The decoding residual data obtained by the inverse orthogonal transformare supplied to the calculation unit 65. To the calculation unit 65, thepredicted image is supplied from the intra prediction unit 71 or themotion prediction/compensation unit 72 through the selection unit 73.

The calculation unit 65 adds the decoding residual data and thepredicted image, thereby providing the decoded image data correspondingto the image data before the predicted image is subtracted by thecalculation unit 23 of the image encoding device 11. The calculationunit 65 supplies the decoded image data to the deblocking filter 31 b.

The deblocking filter 31 b suppresses the block distortion of thedecoded image by performing the deblocking filtering process asappropriate. The deblocking filter 31 b is basically structuredsimilarly to the deblocking filter 31 a of FIG. 1. In other words, thedeblocking filter 31 b includes the strong filter and the weak filter,and which filter to apply is determined using the determinationformulae. The determination formulae of the strong filter include threedetermination formulae, in each of which the Bs value is used inaddition to the parameter beta or tc obtained based on the quantizationparameter QP.

Note that the ON/OFF information of the deblocking filter, which isencoded by the image encoding device 11, and each offset of theparameters beta and tc are received in the image decoding device 51 asthe parameters of the deblocking filter, decoded by the losslessdecoding unit 62, and used by the deblocking filter 31 b.

The deblocking filter 31 b performs the deblocking filtering process onthe image from the calculation unit 30 using the aforementioned ON/OFFinformation, offset, and parameters beta and tc. The deblocking filter31 b supplies the filtering process result to the adaptive offset filter81. Note that the detailed structure of the deblocking filter 31 b isdescribed below with reference to FIG. 14.

The adaptive offset filter 81 performs the offset filter (SAO) processon the image after the filtering with the deblocking filter 31 bespecially for suppressing the ringing.

The adaptive offset filter 81 performs the filtering process on theimage after the filtering with the deblocking filter 31 b using theoffset value for each LCU, for which the type of the offset filter isdecided for each LCU. The adaptive offset filter 81 supplies thefiltered image to the adaptive loop filter 82.

Note that this offset value for each LCU is the value calculated by theadaptive offset filter 41 of the image encoding device 11 and encodedand transmitted as the adaptive offset parameter. The offset value foreach LCU encoded by the image encoding device 11 is received in theimage decoding device 51 as the adaptive offset parameter, decoded bythe lossless decoding unit 62, and used by the adaptive offset filter81.

The adaptive loop filter 82 performs the filtering process on the imageafter the filtering with the adaptive offset filter 81 in units ofprocess using the filter coefficient, and supplies the filtering processresult to the frame memory 69 and the screen rearrangement buffer 67.

Although not illustrated in the example of FIG. 3, in the image decodingdevice 51, the filter coefficient is calculated for each LCU by theadaptive loop filter 42 of the image encoding device 11 and the oneencoded and transmitted as the adaptive loop filter parameter is decodedby the lossless decoding unit 62 and is used.

The screen rearrangement buffer 67 rearranges the images. In otherwords, the order of the frames, which have been rearranged for theencoding by the screen rearrangement buffer 22 of FIG. 1, is rearrangedin the original order of display. The D/A conversion unit 68 performsthe D/A conversion of the image supplied from the screen rearrangementbuffer 67 and outputs the image to the display, which is not shown,where the image is displayed.

The output of the adaptive loop filter 82 is further supplied to theframe memory 69.

The frame memory 69, the selection unit 70, the intra prediction unit71, the motion prediction/compensation unit 72, and the selection unit73 correspond to the frame memory 32, the selection unit 33, the intraprediction unit 34, the motion prediction/compensation unit 35, and thepredicted image selection unit 36 of the image encoding device 11,respectively.

The selection unit 70 reads out the image to be subjected to the interprocess and the image to be referred to out of the frame memory 69, andsupplies the images to the motion prediction/compensation unit 72.Moreover, the selection unit 70 reads out the image used in the intraprediction out of the frame memory 69 and supplies the image to theintra prediction unit 71.

To the intra prediction unit 71, the information representing the intraprediction mode obtained by decoding the header information and the likeare supplied from the lossless decoding unit 62 as appropriate. Based onthis information, the intra prediction unit 71 generates the predictedimage from the reference image acquired from the frame memory 69 andsupplies the generated predicted image to the selection unit 73.

To the motion prediction/compensation unit 72, the pieces of informationobtained by decoding the header information (such as the prediction modeinformation, the motion vector information, the reference frameinformation, the flag, and various parameters) are supplied from thelossless decoding unit 62.

The motion prediction/compensation unit 72 generates the predicted imagefrom the reference image acquired from the frame memory 69 based onthese pieces of information supplied from the lossless decoding unit 62,and supplies the generated predicted image to the selection unit 73.

The selection unit 73 selects the predicted image generated by themotion prediction/compensation unit 72 or the intra prediction unit 71,and supplies the image to the calculation unit 65.

[Operation of Image Decoding Device]

With reference to FIG. 4, description is made of an example of the flowof the decoding process to be executed by the image decoding device 51.

Upon the start of the decoding process, in step S51, the accumulationbuffer 61 receives the transmitted encoded stream (data) and accumulatesthe data. In step S52, the lossless decoding unit 62 decodes the encodeddata supplied from the accumulation buffer 61. I picture, P picture, andB picture encoded by the lossless encoding unit 26 of FIG. 1 aredecoded.

Before the decoding of the pictures, the pieces of information of theparameters such as the motion vector information, the reference frameinformation, the prediction mode information (intra prediction mode orinter prediction mode) are also decoded.

If the prediction mode information is the intra prediction modeinformation, the prediction mode information is supplied to the intraprediction unit 71. If the prediction mode information is the interprediction mode information, the motion vector information or the likecorresponding to the prediction mode information is supplied to themotion prediction/compensation unit 72. Moreover, the parameter of thedeblocking filter and the adaptive offset parameter are decoded andsupplied to the deblocking filter 31 b and the adaptive offset filter81, respectively.

In step S53, the intra prediction unit 71 or the motionprediction/compensation unit 72 performs the predicted image generationprocess in accordance with the prediction mode information supplied fromthe lossless decoding unit 62.

In other words, in the case where the intra prediction mode informationis supplied from the lossless decoding unit 62, the intra predictionunit 71 generates the intra predicted image of the intra predictionmode. In the case where the inter prediction mode information issupplied from the lossless decoding unit 62, the motionprediction/compensation unit 72 performs the motionprediction/compensation process of the inter prediction mode to generatethe inter predicted image.

Through this process, the predicted image (intra predicted image)generated by the intra prediction unit 71 or the predicted image (interpredicted image) generated by the motion prediction/compensation unit 72is supplied to the selection unit 73.

In step S54, the selection unit 73 selects the predicted image. In otherwords, the predicted image generated by the intra prediction unit 71 orthe predicted image generated by the motion prediction/compensation unit72 is supplied. Therefore, the supplied predicted image is selected andsupplied to the calculation unit 65, and is added to the output of theinverse orthogonal transform unit 64 in step S57, which is describedbelow.

In step S52 described above, the transform coefficient decoded by thelossless decoding unit 62 is supplied also to the inverse quantizationunit 63. In step S55, the inverse quantization unit 63 inverselyquantizes the transform coefficient decoded by the lossless decodingunit 62, with the characteristic corresponding to the characteristic ofthe quantization unit 25 of FIG. 1.

In step S56, the inverse orthogonal transform unit 64 performs theinverse orthogonal transform of the transform coefficient inverselyquantized by the inverse quantization unit 63, with the characteristiccorresponding to the characteristic of the orthogonal transform unit 24of FIG. 1. Thus, the differential information corresponding to the inputof the orthogonal transform unit 24 of FIG. 1 (output of the calculationunit 23) is decoded.

In step S57, the calculation unit 65 sums up the differentialinformation and the predicted image selected in the process of theaforementioned step S54 and input through the selection unit 73. Thus,the original image is decoded.

In step S58, the deblocking filter 31 b performs the deblockingfiltering process on the image output from the calculation unit 65. Inthe deblocking filtering process, which is described in detail withreference to FIG. 14, the Bs value is used in the determination relatedto the strong filter. The image after the filtering from the deblockingfilter 31 b is output to the adaptive offset filter 41. In thedeblocking filtering process, the ON/OFF information as the parameter ofthe deblocking filter supplied from the lossless decoding unit 62 andeach offset of the parameters beta and tc are also used.

In step S59, the adaptive offset filter 81 performs the adaptive offsetfiltering process. Through this process, the image after the filteringwith the deblocking filter 31 b is subjected to the filtering processusing the offset value for each LCU for which the type of the offsetfilter is decided for each LCU. The image after the filtering issupplied to the adaptive loop filter 82.

In step S60, the adaptive loop filter 82 performs the adaptive loopfiltering process on the image after the filtering with the adaptiveoffset filter 81. With the use of the filter coefficient calculated inunits of process, the adaptive loop filter 82 performs the filteringprocess in units of process on the input image, and supplies filteringprocess result to the screen rearrangement buffer 67 and the framememory 69.

In step S61, the frame memory 69 stores the filtered image.

In step S62, the screen rearrangement buffer 67 rearranges the imagesafter the adaptive loop filter 82. In other words, the order of frames,which has been rearranged for the encoding by the screen rearrangementbuffer 22 of the image encoding device 11, is rearranged in the originalorder of display.

In step S63, the D/A conversion unit 68 performs the D/A conversion ofthe image from the screen rearrangement buffer 67. This image is outputto the display, which is not shown, where the image is displayed.

Upon the end of the process of step S63, the decoding process ends.

2. First Embodiment Determination Condition of Strong Filter

Next, description is made of determination formulae of the strong filterin the HEVC method and the present technique with reference to FIG. 5.

In the example of FIG. 5, a block P including 4×4 pixels and an adjacentblock Q, which is adjacent to the block P on the right side, are shown.

In the HEVC method, three pixels (hatched rectangles in the drawing)from the boundary between the block P and the adjacent block Q are usedto determine whether to apply the strong filter or not out of thefilters for the boundary therebetween. As the determination formula ofthe strong filter in the HEVC method, the following formulae (1) to (3)are used.[Mathematical Formula 1]|p3i−p0i|+|q3i−q0i|<(beta>>3)  (1)[Mathematical Formula 2]2*(|p2i−2*p1i+p0i|+|q2i−2*q1i+q0i|)<(beta>>2)  (2)[Mathematical Formula 3]|p0i−q0i|<((tc*5+1)>>1)  (3)

In the example of FIG. 5, i=0, 3 in Formulae (1) to (3).

In Formulae (1) and (2), the pixel value in the block is determined.Formulae (1) and (2) indicate that as the block has the gradient values,the strong filter is easily applicable. On the contrary, Formula (3)determines the pixel value at the block boundary.

In other words, if all of Formulae (1) to (3) are true at the blockboundary to be determined, the strong filter is used for that blockboundary; if at least one is false, the determination of the weak filteris performed next.

If the determination formula of the strong filter in the HEVC method isstill used, in the current situation, the weak filter is often appliedwhere the block noise is remarkable, in which case the block noise mayremain.

In view of this, the number of block boundaries for which the strongfilter is applied at the place where the noise is remarkable is desiredto be increased.

In consideration of the above, in the present technique, thedetermination of the strong filter is adjusted so that the strong filteris applied more. For example, the Bs value is used in the determinationof the strong filter as indicated by the next Formulae (4) to (6)instead of Formulae (1) to (3).[Mathematical Formula 4]|p3i−p0i|+|q3i−q0i|<(f1(Bs)*beta>>3)   (4)[Mathematical Formula 5]2*(|p2i−2*p1i+p0i|+|q2i−2*q1i+q0i|)<(f2(Bs)*beta>>2)  (5)[Mathematical Formula 6]|p0i−q0i|<((f3(Bs)*tc*5+1)>>1)  (6)

In the example of FIG. 5, i=0, 3 in Formulae (4) to (6). Moreover, forexample, f1(Bs)=f2(Bs)=f3(Bs)=Bs+2. In the general expression,F(Bs)=a*Bs+b; however, the state of the function does not matter.

Here, the parameters beta and tc, which are used in the determination ofthe strong filter, are obtained by using the quantization parameter QPfrom the table. Thus, as the quantization parameter QP is increased, theparameters beta and tc are increased. Since the larger quantizationparameter tends to increase the quantization noise, these values areused in the determination of the strong filter. In the determination ofthe strong filter, however, the values of the parameters beta and tc aredivided to be smaller and used in the determination; thus, the thresholdis a little smaller in the image where the block noise often appears,such as water or nature.

In view of this, in the present technique, the values of the parametersbeta and tc are increased in the strong filter and the determination isperformed with the larger threshold. Accordingly, Formulae (4) to (6)easily become true and the region where the strong filter is applied canbe increased.

Further, the prediction accuracy in the intra prediction (in-screenprediction) is lower and the prediction error (residual information) islarger than in the inter prediction; thus, the block noise is caused.The Bs value is the parameter presenting the strength for the blockboundary, and is set so that the parameter is larger in the intraprediction than in the inter prediction in consideration of theprediction error.

In other words, the Bs value may be 0, 1, or 2 in the HEVC method. Inthe case of 0, the deblocking filter is not applicable. In the case of1, the prediction mode is not the intra prediction; in the case of 2,the prediction mode is the intra prediction.

In the case of the intra prediction, the threshold can be set larger inthe determination than in the inter prediction by multiplying the Bsvalue as above by the values of the parameters beta and tc. As a result,in the case of the intra prediction, the region where the strong filteris applied can be made larger than in the case of the inter prediction.

As described above, by the use of the Bs value as the threshold in thedetermination of the strong filter, the threshold parameter in the rightside of Formulae (4) to (6) is increased, and the large threshold can beused in the determination. Therefore, Formulae (4) to (6) easily becometrue and the region where the strong filter is applied can be increased.

Note that the description has been made of the example in which the Bsvalue (the linear function of the Bs value) is multiplied by theparameter beta or tc in the threshold parameter in the right side of thedetermination formula for increasing the threshold of the determinationof the strong filter.

Here, the method of increasing the threshold of the determination of thestrong filter is not limited to the multiplication of the Bs value andfor example, the shift value may be decreased in the threshold parameterof the right side of Formulae (1) to (3) in the HEVC method and used.

On this occasion, the shift value is set smaller in the intra predictionthan in the other prediction, and in the other prediction (for example,inter prediction), the shift value is set larger than in the intraprediction.

In other words, the shift value of the determination is changed inaccordance with the prediction mode and used. In this case, the effectsimilar to the effect obtained when the Bs value is used in thethreshold of the determination of the strong filter can be obtained. Inother words, since the shift value is set small in the intra prediction,the threshold in the determination is increased. Therefore, in the caseof the intra prediction, the strong filter is easily applied.

As described above, the prediction error tends to be increased in theintra prediction as compared to the inter prediction, and in this case,the block noise often appears remarkably. Therefore, when the strongfilter is applied in the intra prediction, the block noise can besuppressed further.

Note that the prediction mode is the mode representing the intraprediction or the mode representing the inter prediction. In addition,the prediction mode may include, for example, the vertical mode andhorizontal mode in the intra prediction.

[Detailed Description of Strong Filter Determination]

FIG. 6 represents the relation between the quantization parameter QP andthe value in the right side of Formula (4) when f1(Bs)=Bs+2. In otherwords, in this case, a=1 and b=2 in the linear function of the Bs value.Note that the value in the right side of Formulae (1) and (4) isreferred to as the threshold A.

When the prediction mode is not the intra prediction, Bs value=1; thus,the threshold A in the right side of Formula (4) is (3*beta)>>3. Whenthe prediction mode is the intra prediction, Bs value=2; thus, thethreshold A in the right side of Formula (4) is (4*beta)>>3.

Therefore, for the quantization parameter QP, the threshold A in theright side of Formula (1) in the case of HEVC (conventional technique),the threshold A in the right side of Formula (4) in the case of theprediction other than intra in the present technique, and the thresholdA in the right side of Formula (4) in the case of the intra predictionin the present technique are as shown in FIG. 6.

For example, when the quantization parameter QP is 51, the thresholds Aare as follows: 8 in the case of HEVC; 24 in the case of the predictionother than intra in the present technique; and 32 in the case of theintra prediction in the present technique.

Thus, the threshold A is larger in the case of the prediction other thanintra in the present technique than the threshold A in the case of HEVC.Further, the threshold A in the case of the intra prediction in thepresent technique is larger than the threshold A in the case of theprediction other than intra in the present technique.

FIG. 7 illustrates an example of pixel values of the blocks P and Q ofFIG. 5 in the case of FIG. 6. In the example of FIG. 7, p3i and q3irepresent the pixel values in the range where the strong filter isapplied in the case of HEVC (conventional technique) and p′3i and q′3irepresent the pixel values in the range where the strong filter isapplied in the case of the present technique.

In other words, the value of p′3i−p0i and the value of q′3i−q0i arelarger than the values in the case of HEVC (value of p3i−p0i and valueof q3i−q0i). Thus, the determination formula (Formula (4)) in the caseof the present technique can be made true for the block boundary, whichis false in the determination formula (Formula (1)) in the case of HEVC.

FIG. 8 represents the relation between the quantization parameter QP andthe value in the right side of Formula (5) in the case of f2(Bs)=Bs+2.Note that the values in the right side of Formulae (2) and (5) arereferred to as threshold B.

s When the prediction mode is the prediction other than intra, Bsvalue=1; thus, the threshold B in the right side of Formula (5) is(3*beta)>>2. When the prediction mode is the intra prediction, Bsvalue=2; thus, the threshold B in the right side of Formula (5) is(4*beta)>>2.

Therefore, for the quantization parameter QP, the threshold B in theright side of Formula (2) in the case of HEVC (conventional technique),the threshold B in the right side of Formula (5) in the case of theprediction other than intra in the present technique, and the thresholdB in the right side in the case of Formula (5) in the intra predictionin the present technique are as shown in FIG. 8.

For example, when the quantization parameter QP is 51, the thresholds Bare as follows: 16 in the case of HEVC; 48 in the case of the predictionother than intra in the present technique; and 64 in the case of theintra prediction in the present technique.

In this manner, the threshold B in the case of the prediction other thanintra in the present technique is larger than the threshold B in thecase of HEVC. The threshold B in the case of the intra prediction in thepresent technique is larger than the value in the case of the predictionother than intra in the present technique.

FIG. 9 illustrates an example of the pixel values of the blocks P and Qof FIG. 5 in the case of FIG. 8. In the example of FIG. 9, p2i, phi,q2i, and q1i are the pixel values in the range where the strong filteris applied in the case of HEVC (conventional technique). Moreover, p′2i,p′1i, q′2i, and q′1i are the pixel values in the range where the strongfilter is applied in the case of the present technique.

In other words, as compared to the case of HEVC (value of p2i−p1i andvalue of p1i−p0i, value of q3i−q1i and value of q1i−q0i), the value ofp′2i−p′1i and the value of p′1i−p0i are larger to some extent and thevalue of q′3i−q′1i and the value of q′1i−q0i are larger to some extent.Therefore, the determination formula (Formula (5)) in the case of thepresent technique can be made true for the block boundary for which thedetermination formula (Formula (2)) is false in the case of HEVC.

Formulae (4) and (5) determine the pixel values in the block whileFormula (6) determines the pixel value at the block boundary.

FIG. 10 represents the relation between the quantization parameter QPand the value in the right side of Formula (6) in the case off3(Bs)=Bs+2. The values in the right side of Formula (3) and Formula (6)are referred to as threshold C.

When the prediction mode is the prediction other than intra, Bs value=1;thus, the threshold C in the right side of Formula (6) is (3*tc*5+1)>>1.When the prediction mode is the intra prediction, Bs value=2; thus, thethreshold C in the right side of Formula (6) is (4*tc*5+1)>>1.

Therefore, for the quantization parameter QP, the threshold C in rightside of Formula (3) in the case of HEVC (conventional technique), thethreshold C in the right side of Formula (6) in the case of theprediction other than intra in the present technique, and the thresholdC in the right side in the case of Formula (6) in the intra predictionin the present technique are as shown in FIG. 10.

For example, when the quantization parameter QP is 53, the threshold Cin the case of the prediction other than intra in HEVC is 50. In HEVC,however, in the case of the intra prediction, 2 is added to the value ofQP for searching tc out of the table; thus, the threshold C in the caseof the intra prediction in HEVC is 60 though not illustrated in FIG. 10.On the other hand, the threshold C in the case of the prediction otherthan intra in the present technique is 150 and the threshold C in thecase of the intra prediction in the present technique is 200.

Thus, the threshold C in the case of the prediction other than intra inthe present technique is larger than the threshold C in the case ofHEVC. Moreover, the threshold C in the case of the intra prediction inthe present technique is larger than the threshold C in the case of theprediction other than intra in the present technique.

Note that the value of tc is also used in the portion (filteringprocess) other than the strong/weak filter determination in thedeblocking filter, and in the present technique, the product ofmultiplying tc by the Bs value is used only in the strong/weak filterdetermination.

FIG. 11 illustrates the example of the pixel values of the blocks P andQ of FIG. 5 in the case of FIG. 10. In the example of FIG. 11, p0i andq0i represent the pixel values in the range where the strong filter isapplied in the case of HEVC (conventional technique). Moreover, p′0i andq′0i represent the pixel values in the range where the strong filter isapplied in the case of the present technique.

In other words, as compared to the case of HEVC (value p0i−q0i), thevalue of p′0i−q′0i is larger to some extent. Therefore, thedetermination formula (Formula (6)) in the case of the present techniquecan be made true for the block boundary for which the determinationformula (Formula (3)) is false in the case of HEVC.

As described thus, Formulae (4) to (6) are highly likely to be true ascompared to the case of HEVC. Thus, in the present technique, the strongfilter can be applied at the block boundary where the strong filter hasbeen unable to be applied in the case of HEVC, i.e., in the region wherethe blocking noise is remarkable.

Specific Example

Next, a specific example of the HEVC (conventional technique) and thepresent technique is described with reference to FIG. 12.

The pixel values at the block boundary before the deblocking filter are120, 121, 121, 119, 147, 144, 142, and 140 from the left. In the exampleof FIG. 12, the block boundary is present between the pixel value 119and the pixel value 147.

For example, when the quantization parameter QP is 37, the value of theleft side of Formulae (1) and (4) is:d=|120−119|+|147−140|=8

When the quantization parameter QP is 37, the threshold A in the rightside of Formula (1) is 4 and the threshold A in the right side ofFormula (4) is 18; thus, Formula (1) is false in the case of HEVC, butFormula (4) is true in the case of the present technique.

Similarly, when the quantization parameter QP is 37, the value of theleft side of Formulae (2) and (5) is:d=2*(|121−2*121+121|+|147−2*144+142|=6

Here, when the quantization parameter QP is 37, the threshold B in theright side of Formula (2) is 9 and the threshold B in the right side ofFormula (5) is 36; thus, Formula (2) is true in the case of HEVC andFormula (5) is true in the case of the present technique.

Moreover, when the quantization parameter QP is 37, the value in theleft side of Formulae (3) and (6) is:d=|147−119|=38

Here, when the quantization parameter QP is 37, the threshold C in theright side of Formula (3) is 10 and the threshold C in the right side ofFormula (6) is 40; thus, Formula (3) is true in the case of HEVC andFormula (6) is true in the case of the present technique.

Thus, in the case of HEVC, the strong filter is not applied at the blockboundary, but the weak filter is used, for example. On the other hand,in the case of the present technique, the strong filter is applied atthis block boundary.

As a result, the pixel values after the deblocking (weak) filter in thecase of HEVC are 120, 121, 124, 126, 140, 141, 192, and 140 from theleft. The pixel values at the block boundary are 126 and 140, and thereare still a lot of differences in pixel value between the blocks.

On the other hand, the pixel values after the deblocking (strong) filterin the case of the present technique are 120, 124, 127, 130, 135, 138,139, and 140 from the left. The pixel values at the block boundary are130 and 135, and there is little difference in pixel value between theblocks. Thus, the block noise can be made less visible.

For example, the natural motion such as the water flow in the river orthe quivering of leaves is hard to predict and the intra prediction ismore suitable; thus, the present technique is particularly effective forthose kinds of images.

[Structure Example of Deblocking Filter]

Next, detailed description is made of the deblocking filter 31 a in theimage encoding device of FIG. 1 and the deblocking filter 31 b in theimage decoding device of FIG. 3. Note that since the deblocking filter31 a in the image encoding device of FIG. 1 and the deblocking filter 31b in the image decoding device of FIG. 3 have the basically similarstructure and operation, the filters are described as the deblockingfilter 31 collectively.

The deblocking filter 31 a and the deblocking filter 31 b are differentin the following point. In the case of the deblocking filter 31 a, theON/OFF information as to whether the deblocking filter is applied or notand the offsets of the parameters beta and tc are input through theoperation unit, which is not shown. On the other hand, in the case ofthe deblocking filter 31 b, the ON/OFF information as to whether thedeblocking filter is applied or not and the offsets of the parametersbeta and tc, which have been encoded in the image encoding device 11,are received and decoded in the lossless decoding unit 62 and input.

FIG. 13 is a block diagram illustrating a structure example of thedeblocking filter.

In the example of FIG. 13, the deblocking filter 31 includes an imagememory 101, a block boundary determination unit 102, a filter strengthdetermination unit 103, a filter calculation unit 104, a coefficientmemory 105, and a control unit 106.

The image memory 101 is formed by a line memory, and stores the imagedata supplied from an addition unit 30. The image memory 101 reads outthe stored image data and supplies the data to the block boundarydetermination unit 102, the filter strength determination unit 103, andthe filter calculation unit 104.

At other than the line boundary, the image data may not be stored in theimage memory 101 and the image data supplied from the addition unit 30may be supplied to each unit and processed. In the example of FIG. 13,however, the image data through the image memory 101 are processed forthe convenience of description.

The block boundary determination unit 102 derives the boundary for everyeight lines and calculates the parameter used for the determinationunder the control of the control unit 106, thereby determining theboundary between the blocks for every four lines. In other words, theblock boundary determination unit 102 derives the boundary between TUand PU using the image data read out of the image memory 101, therebyderiving the Bs value. Moreover, the block boundary determination unit102 calculates the average QP (quantization parameter) by averaging thequantization parameters QP in two regions adjacent to the boundary to beprocessed, and thus the parameters tc and beta are calculated based onthe calculated average QP.

Then, the block boundary determination unit 102 determines whether toperform filtering or not for every four lines by using the image data oftwo blocks adjacent to each other with the block boundary interposedtherebetween out of the image memory 101 and the calculated parameter.The block boundary determination unit 102 supplies the calculatedparameter with the boundary determination result to the filter strengthdetermination unit 103.

The filter strength determination unit 103 determines the filterstrength for every four lines under the control of the control unit 106.In other words, if the block boundary determination unit 102 hasdetermined to perform the filtering, the filter strength determinationunit 103 determines at which strength of the strong filter or the weakfilter the filtering process is performed, and outputs the determinationresult to the filter calculation unit 104. On this occasion, the filterstrength determination unit 103 uses the Bs value in the determinationformula of the strong filter.

The filter calculation unit 104 performs the filter calculation at thefilter strength determined by the filter strength determination unit 103for every four lines by using the image data stored in the image memory101 and the filter coefficient read out of the coefficient memory 105under the control of the control unit 106. The filter calculation unit104 outputs the image data, for which the filtering process has beenperformed, to the frame memory 32 in the later stage.

The coefficient memory 105 stores the filter coefficient used in thefilter calculation of the deblocking filtering process. The coefficientmemory 105 reads out the stored filter coefficient, and supplies thecoefficient to the filter calculation unit 104.

The control unit 106 inputs the information (such as the ON/OFFinformation of the deblocking filter, the offset values of theparameters beta and tc, and the parameter in the clip process) from theoperation unit (lossless decoding unit 62 in the case of the decodingside), which is not shown. To the control unit 106, the parameternecessary for the deblocking filter such as the prediction modeinformation or the quantization parameter is supplied.

For example, the control unit 106 supplies the input information to thecorresponding unit or controls each unit of the deblocking filter 31based on the input ON/OFF information.

The control unit 106 stores the image data for several predeterminedlines on the lower end side in the block and reads out the stored imagedata by controlling the image memory 101. The control unit 106 controlsthe filter strength determination unit 103 to use the Bs value in thedetermination of the strong filter. Alternatively, the control unit 106controls the filter strength determination unit 103 and changes themagnitude of the shift value in accordance with the prediction mode.

[Operation of Deblocking Filter Processing Unit]

Next, the deblocking process of the deblocking filter 31 of FIG. 13 isdescribed with reference to the flowchart of FIG. 14. Note that theprocess includes the process of step S22 of FIG. 2 or step S58 of FIG.4.

For example, the ON/OFF information, the value of the beta offset, thevalue of the tc offset, and the parameter in the clip process are inputto the control unit 106 through the operation unit (lossless decodingunit 62 in the case of the decoding side), which is not shown. Moreover,to the control unit 106, the parameter necessary for the deblockingfilter, such as the prediction mode information or the quantizationparameter is supplied.

In step S101, the control unit 106 sets the offset of the filter (betaoffset and tc offset) and supplies the set offset information to thefilter strength determination unit 103.

In step S102, the control unit 106 determines whether the deblockingfilter is unavailable or not based on the ON/OFF information. If it hasbeen determined in step S102 that the deblocking filter is unavailable,the deblocking filtering process ends.

If it has been determined in step S102 that the deblocking filter is notunavailable, the control unit 106 notifies this to the block boundarydetermination unit 102, the filter strength determination unit 103, andthe filter calculation unit 104, and the process advances to step S103.On this occasion, the parameter necessary for each unit is also suppliedfrom the control unit 106.

In step S103, the block boundary determination unit 102 derives theboundary between TU and PU in units of eight lines. In step S104, theblock boundary determination unit 102 derives the Bs value based on theinformation such as the boundary between TU and PU derived in step S103and the prediction mode information from the control unit 106.

In step S105, the block boundary determination unit 102, the filterstrength determination unit 103, and the filter calculation unit 104perform the filtering process for the luminance boundary. In thefiltering process for the luminance boundary, the filtering for theluminance boundary is performed on the luminance signal by the processof step S105 as described below with reference to FIG. 15.

In step S106, the block boundary determination unit 102, the filterstrength determination unit 103, and the filter calculation unit 104perform the filtering process for the color difference boundary. In theprocess of step S106, the filtering for the color difference boundary isperformed on the color difference signal.

In step S107, the control unit 106 determines whether all the boundarieshave been processed. If it has been determined in step S107 that theprocess of all the boundaries is not finished yet, the process returnsto step S105 and the processes of step S105 and subsequent thereto arerepeated. If it has been determined in step S107 that all the boundariesare already processed in step S107, the process advances to step S108.

In step S108, the control unit 106 determines whether all the CUs havebeen processed or not. If it has been determined in step S108 that theprocess of all the CUs is not finished yet, the process returns to stepS103 and the processes of step S103 and subsequent thereto are repeated.

If it has been determined in step S108 that all the CUs are processed,the deblocking filtering process ends.

[Example of Filtering Process for Luminance Boundary]

Next, the filtering process for the luminance boundary in step S105 inFIG. 14 is described with reference to the flowchart of FIG. 15.

The block boundary determination unit 102 determines in step S131whether the Bs value is larger than 0 or not in units of eight lines. Ifit has been determined in step S131 that the Bs value is not larger than0, the filtering process for luminance boundary ends. In other words,the filtering process by the filter calculation unit 104 is notperformed on the luminance boundary in this case and the pixel valuebefore the filtering is output.

If it has been determined in step S131 that the Bs value is larger than0, the process advances to step S132. In step S132, the block boundarydetermination unit 102 calculates the average QP (quantizationparameter) by averaging the quantization parameters QP in the tworegions adjacent to the boundary to be processed with the use of thepixel values from the image memory 101 in units of eight lines. Thequantization parameter QP is supplied from the control unit 106.

In step S133, the block boundary determination unit 102 calculates theparameters tc and beta based on the average QP calculated by step S132in units of eight lines.

In step S134, the block boundary determination unit 102 performs theon/off determination of the filter in units of four lines. In otherwords, the block boundary determination unit 102 performs the on/offdetermination of the filter in units of four lines by using thecalculated parameters beta or tc, or the like.

The on/off determination result in step S134 is supplied to the filterstrength determination unit 103 together with the calculated parameters(beta, tc, Bs values) as the boundary determination result.

In step S135, the filter strength determination unit 103 sets thethreshold of the determination of the strong filter using each parameterunder the control of the control unit 106. In step S135, for example, inthe case of using the Bs value, the threshold in the determination inFormulae (4) to (6) is set in accordance with the linear function of theBs value from the control unit 106. When the shift value is changed, theshift value in Formulae (1) to (3) is changed in accordance with theprediction mode from the control unit 106 and the threshold in thedetermination is set.

In step S136, the filter strength determination unit 103 determineswhether to apply the strong filter in units of four lines, by usingFormulae (4) to (6) in which the threshold is set. If it has beendetermined in step S136 that Formulae (4) to (6) are true and the strongfilter is applied, the process advances to step S137.

The on/off determination result in step S134 and the strong filterdetermination result in step S137 are supplied to the filter calculationunit 104. On this occasion, the parameter tc is also supplied to thefilter calculation unit 104.

In step S137, the filter calculation unit 104 performs the strongfiltering process for four lines to be processed, based on thedetermination result from the filter strength determination unit 103.The filter calculation unit 104 outputs the pixel value after thefiltering process to the later stage.

If it has been determined in step s136 to apply the strong filter, theprocess advances to step S138. In step S138, the filter strengthdetermination unit 103 determines whether to apply the weak filter inunits of four lines. If it has been determined in step S138 to apply theweak filter, the process advances to step S139.

The on/off determination result in step S134 and the weak filterdetermination result in step S138 are supplied to the filter calculationunit 104. On this occasion, the parameter tc is also supplied to thefilter calculation unit 104.

In step S139, the filter calculation unit 104 uses the weak filter forthe four lines to be processed, based on the determination result fromthe filter strength determination unit 103. The filter calculation unit104 outputs the pixel value after the filtering process to the laterstage.

If it has been determined in step S138 not to use the weak filter, theprocess advances to step S140. In this case, the filter calculation unit104 outputs the pixel value after the filtering process to the laterstage without any process for four lines to be processed, based on thedetermination result from the filter strength determination unit 103.

In step S140, the filter calculation unit 104 determines whether theprocess for eight lines has been completed or not, and if it has beendetermined that the process for the eight lines is completed, thedeblocking of the luminance signal ends. If it has been determined instep S140 that the process for the eight lines is not yet completed, theprocess returns to step S134 and the processes of step S134 andsubsequent thereto are repeated.

Thus, by using the Bs value or by decreasing the shift value, thedetermination of the strong filter is performed; therefore, thethreshold of the determination becomes larger and the strong filter isapplied at more boundaries. Thus, in the deblocking process, the strongfilter can be applied as appropriate.

In the above description, the HEVC method is used basically as theencoding method. The present disclosure is, however, not limited theretoand other encoding method/decoding method including at least thedeblocking filter as the in-loop filter can be used.

Note that the present disclosure can be applied to an image encodingdevice and an image decoding device that are used when the imageinformation (bit stream) compressed by the orthogonal transform, such asthe discrete cosine transform, and the motion compensation is receivedvia a network medium such as the satellite broadcasting, the cabletelevision, the Internet, or the cellular phone, as in the HEVC methodor the like, for example. Moreover, the present disclosure can beapplied to an image encoding device and an image decoding device thatare used in the process in a storage medium such as an optical ormagnetic disc or a flash memory.

3. Second Embodiment Application to Multi-Viewpoint ImageEncoding/Multi-Viewpoint Image Decoding

The aforementioned processes can be applied to the multi-viewpoint imageencoding/multi-viewpoint image decoding. FIG. 16 illustrates an exampleof the multi-viewpoint image encoding method.

As illustrated in FIG. 16, the multi-viewpoint image includes images ofa plurality of viewpoints, and the image of a predetermined viewpointamong the plurality of viewpoints is specified as the image of the baseview. The images of the viewpoints other than the image of the base vieware treated as the non-base view images.

In encoding the multi-viewpoint image as illustrated in FIG. 16, theparameter of the deblocking filter is set in each view (same view), andwhether to apply the strong filter or not can be determined by using thestrength parameter of the deblocking filter. In other words, theparameter of the deblocking filter set in another view or thedetermination result of the strong filter can be used in common in eachview (different view).

In this case, the parameter of the deblocking filter set in the baseview or whether to apply the strong filter determined is used in atleast one non-base view.

Alternatively, the parameter of the deblocking filter set in thenon-base view (view_id=i) or whether to apply the strong filterdetermined is used in at least one of the base view and the non-baseview (view_id=j).

Thus, the strong filter can be used as appropriate in the deblockingprocess. As a result, the block noise can be suppressed.

[Multi-Viewpoint Image Encoding Device]

FIG. 17 is a diagram illustrating a multi-viewpoint image encodingdevice for the multi-viewpoint image encoding described above. Asillustrated in FIG. 17, a multi-viewpoint image encoding device 600includes an encoding unit 601, an encoding unit 602, and a multiplexingunit 603.

The encoding unit 601 encodes the base view image to generate thebase-view image encoded stream. The encoding unit 602 encodes thenon-base view image to generate the non-base view image encoded stream.The multiplexing unit 603 multiplexes the base view image encoded streamgenerated in the encoding unit 601 and the non-base view image encodedstream generated in the encoding unit 602, thereby generating themulti-viewpoint image encoded stream.

As the encoding unit 601 and the encoding unit 602 of themulti-viewpoint image encoding device 600, the image encoding device 11(FIG. 1) can be applied. In this case, the multi-viewpoint imageencoding device 600 sets and transmits the parameter of the deblockingfilter set by the encoding unit 601 and the parameter of the deblockingfilter set by the encoding unit 602.

The parameter of the deblocking filter set by the encoding unit 601 maybe set and transmitted to be commonly used in the encoding unit 601 andthe encoding unit 602. On the contrary, the parameter of the deblockingfilter set by the encoding unit 602 may be set and transmitted to becommonly used in the encoding unit 601 and the encoding unit 602.

[Multi-Viewpoint Image Decoding Device]

FIG. 18 is a diagram illustrating a multi-viewpoint image decodingdevice for the aforementioned multi-viewpoint image decoding. Asillustrated in FIG. 18, a multi-viewpoint image decoding device 610includes a demultiplexing unit 611, a decoding unit 612, and a decodingunit 613.

The demultiplexing unit 611 demultiplexes the multi-viewpoint imageencoded stream, in which the base view image encoded stream and thenon-base view image encoded stream are multiplexed, thereby extractingthe base view image encoded stream and the non-base view image encodedstream. The decoding unit 612 decodes the base view image encoded streamextracted by the demultiplexing unit 611, thereby providing the baseview image. The decoding unit 613 decodes the non-base view imageencoded stream extracted by the demultiplexing unit 611, therebyproviding the non-base view image.

As the decoding unit 612 and the decoding unit 613 of themulti-viewpoint image decoding device 610, the multi-viewpoint imagedecoding device 51 (FIG. 3) can be applied. In this case, themulti-viewpoint image decoding device 610 performs the process using theparameter of the deblocking filter set by the encoding unit 601 anddecoded by the decoding unit 612, and the parameter of the deblockingfilter set by the encoding unit 602 and decoded by the decoding unit613.

Note that the parameter of the deblocking filter set by the encodingunit 601 (or encoding unit 602) may be set and transmitted to becommonly used in the encoding unit 601 and the encoding unit 602. Inthis case, the multi-viewpoint image decoding device 610 performs theprocess by using the parameter of the deblocking filter set by theencoding unit 601 (or encoding unit 602) and decoded by the decodingunit 612 (or decoding unit 613).

4. Third Embodiment Application to Layer Image Encoding/Layer ImageDecoding

The aforementioned processes can be applied to the layer imageencoding/layer image decoding. FIG. 19 illustrates an example of themulti-viewpoint image encoding method.

As illustrated in FIG. 19, the layer image includes images of aplurality of layers (resolutions), and the image of a predeterminedlayer among the plurality of resolutions is specified as the image ofthe base layer. The images of the layers other than the base layer imageare treated as the non-base layer images.

In the case of the layer image encoding (spatial scalability) asillustrated in FIG. 19, the parameter of the deblocking filter is setand whether to apply the strong filter can be determined by using thestrength parameter of the deblocking filter. Moreover, among the layers(different layers), the parameters of the deblocking filter set in theother layers or the determination results of the strong filter can beshared.

In this case, the parameter of the deblocking filter set in the baselayer or whether to apply the strong filter determined is used in atleast one non-base layer.

Alternatively, the parameter of the deblocking filter set in thenon-base layer (layer_id=i) or whether to apply the strong filterdetermined is used in at least one of the base layer and the non-baselayer (layer_id=j).

Thus, the strong filter can be applied in the deblocking process asappropriate. As a result, the block noise can be suppressed.

[Layer Image Encoding Device]

FIG. 20 is a diagram illustrating a layer image encoding device for theaforementioned layer image encoding. As illustrated in FIG. 20, thelayer image encoding device 620 includes an encoding unit 621, anencoding unit 622, and a multiplexing unit 623.

The encoding unit 621 encodes the base layer image to generate the baselayer image encoded stream. The encoding unit 622 encodes the non-baselayer image to generate the non-base layer image encoded stream. Themultiplexing unit 623 multiplexes the base layer image encoded streamgenerated in the encoding unit 621 and the non-base layer image encodedstream generated in the encoding unit 622, thereby generating the layerimage encoded stream.

As the encoding unit 621 and the encoding unit 622 in the layer imageencoding device 620, the image encoding device 11 (FIG. 1) can beapplied. In this case, the layer image encoding device 620 sets andtransmits the parameter of the deblocking filter set by the encodingunit 621 and the parameter of the deblocking filter set by the encodingunit 602.

As described above, the parameter of the deblocking filter set by theencoding unit 621 may be set and transmitted to be commonly used in theencoding unit 621 and the encoding unit 622. On the contrary, theparameter of the deblocking filter set by the encoding unit 622 may beset and transmitted to be commonly used in the encoding unit 621 and theencoding unit 622.

[Layer Image Decoding Device]

FIG. 21 is a diagram illustrating a layer image decoding device for theaforementioned layer image decoding. As illustrated in FIG. 21, a layerimage decoding device 630 includes a demultiplexing unit 631, a decodingunit 632, and a decoding unit 633.

The demultiplexing unit 631 demultiplexes the layer image encodedstream, in which the base layer image encoded stream and the non-baselayer image encoded stream are multiplexed, thereby extracting the baselayer image encoded stream and the non-base layer image encoded stream.The decoding unit 632 decodes the base layer image encoded streamextracted by the demultiplexing unit 631, thereby providing the baselayer image. The decoding unit 633 decodes the non-base layer imageencoded stream extracted by the demultiplexing unit 631, therebyproviding the non-base layer image.

As the decoding unit 632 and the decoding unit 633 of this layer imagedecoding device 630, the multi-viewpoint image decoding device 51 (FIG.3) can be applied. In this case, the layer image decoding device 630performs the process by using the parameter of the deblocking filter setby the encoding unit 621 and decoded by the decoding unit 632, and theparameter of the deblocking filter set by the encoding unit 622 anddecoded by the decoding unit 633.

Note that the parameter of the deblocking filter set by the encodingunit 621 (or encoding unit 622) may be set and transmitted to becommonly used in the encoding unit 621 and the encoding unit 622. Inthis case, the layer image decoding device 630 performs the process byusing the parameter of the deblocking filter set by the encoding unit621 (or encoding unit 622) and decoded by the decoding unit 632 (ordecoding unit 633).

5. Fourth Embodiment Structure Example of Computer

The aforementioned processes can be executed using either hardware orsoftware. In the case of executing the processes with the software,programs constituting the software are installed in a computer. Here,the computer includes a computer incorporated in dedicated hardware or aversatile personal computer capable of executing various functions byhaving various programs installed therein.

FIG. 22 is a block diagram illustrating a structure example of thehardware of the computer executing the aforementioned processes with theprograms.

In a computer 800, a CPU (Central Processing Unit) 801, a ROM (Read OnlyMemory) 802, and a RAM (Random Access Memory) 803 are connected to eachother with a bus 804.

The bus 804 further has an input/output interface 805 connected thereto.The input/output interface 805 also has an input unit 806, an outputunit 807, a storage unit 808, a communication unit 809, and a drive 810connected thereto.

The input unit 806 corresponds to, for example, a keyboard, a mouse, amicrophone, or the like. The output unit 807 corresponds to, forexample, a display, a speaker, or the like. The storage unit 808corresponds to, for example, a hard disk, a nonvolatile memory, or thelike. The communication unit 809 corresponds to, for example, a networkinterface. The drive 810 drives a removable recording medium 811 such asa magnetic disk, an optical disk, a magneto-optic disk, or asemiconductor memory.

In the computer with the above structure, the CPU 801 loads the programsstored in the storage unit 808 to the RAM 803 through the input/outputinterface 805 and the bus 804 and executes the programs, therebyperforming the abovementioned processes.

The programs executed by the computer 800 (CPU 801) can be provided bybeing recorded in, for example, the removable recording medium 811 as apackage medium, and supplied. The programs can be provided through thewired or wireless transmission medium such as the local area network,the Internet, or digital satellite broadcasting.

In the computer, the programs can be installed in the storage unit 808through the input/output interface 805 by having the removable recordingmedium 811 attached to the drive 810. Alternatively, the programs can beinstalled in the storage unit 808 by having the programs received in thecommunication unit 809 through a wired or wireless transmission medium.Further alternatively, the programs can be installed in advance in theROM 802 or the storage unit 808.

The programs to be executed by the computer may be the programs thatenable the process in time series order as described in thisspecification or that enable the processes in parallel or at necessarytiming such as when the calling is made.

In this specification, the steps describing the program recorded in therecording medium include not just the process performed in the timeseries order as described but also the process that is not necessarilyperformed in the time series but executed in parallel or individually.

In this specification, the system refers to an entire device including aplurality of devices (pieces of apparatus).

In the above description, the structure described as one device (orprocessing unit) may be divided into and structured as a plurality ofdevices (or processing units). On the contrary, the structure describedabove as the plural devices (or processing units) may be structured asone device (or processing unit). The structure other than thosedescribed above may be added to the structure of each device (orprocessing unit). As long as the structure or operation as the wholesystem is substantially the same, a part of the structure of a certaindevice (or processing unit) may be included in a structure of anotherdevice (or processing unit). Thus, the present technique is not limitedto the above embodiment and various modifications can be made withoutdeparting from the scope of the present technique.

The image encoding device and image decoding device according to theabove embodiments can be applied to various electronic appliancesincluding a transmitter or a receiver used in the distribution on thesatellite broadcasting, wired broadcasting such as cable TV, or theInternet, or the distribution to the terminal through the cellularcommunication, a recording device that records the images in a mediumsuch as an optical disk, a magnetic disk, or a flash memory, and areproducing device that reproduces the image from these storage media.Description is hereinafter made of four application examples.

6. Application Examples First Application Example: Television Receiver

FIG. 23 illustrates an example of a schematic structure of a televisiondevice to which the above embodiment has been applied. A televisiondevice 900 includes an antenna 901, a tuner 902, a demultiplexer 903, adecoder 904, a video signal processing unit 905, a display unit 906, anaudio signal processing unit 907, a speaker 908, an external interface909, a control unit 910, a user interface 911, and a bus 912.

The tuner 902 extracts a signal of a desired channel from broadcastingsignals received through the antenna 901, and demodulates the extractedsignal. The tuner 902 outputs an encoded bit stream obtained by thedemodulation to the demultiplexer 903. In other words, the tuner 902 hasa role of a transmission unit in the television device 900 for receivingthe encoded stream in which the image is encoded.

The demultiplexer 903 separates the video stream and the audio stream ofthe program to be viewed from the encoded bit stream, and outputs theseparated streams to the decoder 904. The demultiplexer 903 extracts anauxiliary piece of data such as EPG (Electronic Program Guide) from theencoded bit stream, and supplies the extracted data to the control unit910. Note that the demultiplexer 903 may descramble the encoded bitstream if the encoded bit stream has been scrambled.

The decoder 904 decodes the video stream and the audio stream input fromthe demultiplexer 903. The decoder 904 outputs the video data generatedby the decoding process to the video signal processing unit 905. Thedecoder 904 moreover outputs the audio data generated by the decodingprocess to the audio signal processing unit 907.

The video signal processing unit 905 reproduces the video data inputfrom the decoder 904, and displays the video on the display unit 906.The video signal processing unit 905 may display the application screensupplied through the network on the display unit 906. The video signalprocessing unit 905 may perform an additional process such as noiseremoval (suppression) on the video data in accordance with the setting.Moreover, the video signal processing unit 905 may generate the image ofGUI (Graphical User Interface) such as a menu, a button, or a cursor andoverlap the generated image on the output image.

The display unit 906 is driven by a drive signal supplied from the videosignal processing unit 905, and displays the video or image on the videoscreen of a display device (such as a liquid crystal display, a plasmadisplay, or an GELD (Organic ElectroLuminescence Display) (organic ELdisplay)).

The audio signal processing unit 907 performs the reproduction processsuch as D/A conversion or amplification on the audio data input from thedecoder 904, and outputs the audio from the speaker 908. Moreover, theaudio signal processing unit 907 may perform the additional process suchas noise removal (suppression) on the audio data.

The external interface 909 is the interface for connecting between thetelevision device 900 and an external appliance or a network. Forexample, the video stream or audio stream received through the externalinterface 909 may be decoded by the decoder 904. In other words, theexternal interface 909 also has a role of a transmission unit in thetelevision device 900 for receiving the encoded stream in which theimage is encoded.

The control unit 910 includes a processor such as a CPU, and a memorysuch as a RAM and a ROM. The memory stores programs to be executed bythe CPU, program data, EPG data, and data acquired through the network,etc. The programs stored in the memory are read in and executed by theCPU when the television device 900 is activated, for example. Byexecuting the programs, the CPU controls the operation of the televisiondevice 900 in response to an operation signal input from the userinterface 911, for example.

The user interface 911 is connected to the control unit 910. The userinterface 911 includes, for example, a button and a switch for a user tooperate the television device 900, and a reception unit for receiving aremote control signal. The user interface 911 generates the operationsignal by detecting the operation of the user through these components,and outputs the generated operation signal to the control unit 910.

The bus 912 connects among the tuner 902, the demultiplexer 903, thedecoder 904, the video signal processing unit 905, the audio signalprocessing unit 907, the external interface 909, and the control unit910.

In the television device 900 with the above structure, the decoder 904has a function of the image decoding device according to the aboveembodiment. Thus, in the decoding of the image in the television device900, the block noise can be suppressed.

Second Application Example: Cellular Phone

FIG. 24 illustrates an example of a schematic structure of a cellularphone to which the above embodiment has been applied. A cellular phone920 includes an antenna 921, a communication unit 922, an audio codec923, a speaker 924, a microphone 925, a camera unit 926, an imageprocessing unit 927, a multiplexing/separating unit 928, arecording/reproducing unit 929, a display unit 930, a control unit 931,an operation unit 932, and a bus 933.

The antenna 921 is connected to the communication unit 922. The speaker924 and the microphone 925 are connected to the audio codec 923. Theoperation unit 932 is connected to the control unit 931. The bus 933connects among the communication unit 922, the audio codec 923, thecamera unit 926, the image processing unit 927, themultiplexing/separating unit 928, the recording/reproducing unit 929,the display unit 930, and the control unit 931.

The cellular phone 920 performs the operations including the exchange ofaudio signals, email, and image data, the photographing of images, andthe recording of the data in various modes including the voice callingmode, the data communication mode, the photographing mode, and the videocalling mode.

In the voice calling mode, the analog audio signal generated by themicrophone 925 is supplied to the audio codec 923. The audio codec 923converts the analog audio signal into the audio data, and compresses theconverted audio data through the A/D conversion. Then, the audio codec923 outputs the compressed audio data to the communication unit 922. Thecommunication unit 922 encodes and modulates the audio data andgenerates a transmission signal. The communication unit 922 transmitsthe generated transmission signal to a base station (not shown) throughthe antenna 921. The communication unit 922 amplifies the wirelesssignal received through the antenna 921 and converts the frequencythereof, and acquires the reception signal. The communication unit 922then generates the audio data by demodulating and decoding the receptionsignal, and outputs the generated audio data to the audio codec 923. Theaudio codec 923 extends the audio data and performs the D/A conversionthereon, and generates the analog audio signal. The audio codec 923supplies the generated audio signal to the speaker 924 to output theaudio.

In the data communication mode, for example, the control unit 931generates the text data constituting the email in response to the useroperation through the operation unit 932. The control unit 931 displaysthe text on the display unit 930. The control unit 931 generates theemail data in response to the transmission instruction from the userthrough the operation unit 932, and outputs the generated email data tothe communication unit 922. The communication unit 922 encodes andmodulates the email data, and generates the transmission signal. Thecommunication unit 922 transmits the generated transmission signal tothe base station (not shown) through the antenna 921. The communicationunit 922 amplifies the wireless signal received through the antenna 921and converts the frequency thereof, and acquires the reception signal.The communication unit 922 then decompresses the email data bydemodulating and decoding the reception signal, and outputs thedecompressed email data to the control unit 931. The control unit 931causes the display unit 930 to display the content of the email, and atthe same time, stores the email data in the storage medium of therecording/reproducing unit 929.

The recording/reproducing unit 929 has an arbitrary readable andwritable storage medium. For example, the storage medium may be abuilt-in type storage medium such as a RAM or a flash memory, or adetachable storage medium such as a hard disk, a magnetic disk, amagneto-optic disk, and an optical disk, a USB (Universal Serial Bus)memory, or a memory card.

In the photographing mode, for example, the camera unit 926 photographsa subject, generates the image data, and outputs the generated imagedata to the image processing unit 927. The image processing unit 927encodes the image data input from the camera unit 926, and stores theencoded stream in the storage medium of the recording/reproducing unit929.

In the video calling mode, for example, the multiplexing/separating unit928 multiplexes the video stream encoded by the image processing unit927 and the audio stream input from the audio codec 923, and outputs themultiplexed stream to the communication unit 922. The communication unit922 encodes and modulates the stream and generates the transmissionsignal. Then, the communication unit 922 transmits the generatedtransmission signal to a base station (not shown) through the antenna921. Moreover, the communication unit 922 amplifies the wireless signalreceived through the antenna 921 and converts the frequency thereof, andacquires the reception signal. These transmission signal and receptionsignal can include the encoded bit stream. Then, the communication unit922 decompresses the stream by demodulating and decoding the receptionsignal, and outputs the decompressed stream to themultiplexing/separating unit 928. The multiplexing/separating unit 928separates the video stream and the audio stream from the input stream,and outputs the video stream to the image processing unit 927 and theaudio stream to the audio codec 923. The image processing unit 927decodes the video stream and generates the video data. The video dataare supplied to the display unit 930 where a series of images aredisplayed. The audio codec 923 extends the audio stream and performs theD/A conversion thereon, and generates the analog audio signal. The audiocodec 923 supplies the generated audio signal to the speaker 924 tooutput the audio.

In the cellular phone 920 with the above structure, the image processingunit 927 has a function of the image encoding device and the imagedecoding device according to the above embodiment. Thus, in the encodingand decoding of the image in the cellular phone 920, the block noise canbe suppressed.

Third Application Example: Recording/Reproducing Device

FIG. 25 illustrates an example of a schematic structure of arecording/reproducing device to which the above embodiment has beenapplied. A recording/reproducing device 940 encodes the audio data andthe video data of the received broadcast program, and records the datain the recording medium, for example. The recording/reproducing device940 may encode the audio data and the video data acquired from anotherdevice, and record the data in the recording medium. Therecording/reproducing device 940 reproduces the data recorded in therecording medium on the monitor and speaker in response to the userinstruction. In this case, the recording/reproducing device 940 decodesthe audio data and the video data.

The recording/reproducing device 940 includes a tuner 941, an externalinterface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a diskdrive 945, a selector 946, a decoder 947, an OSD (On-Screen Display)948, a control unit 949, and a user interface 950.

The tuner 941 extracts a signal of a desired channel from broadcastingsignals received through an antenna (not shown), and demodulates theextracted signal. The tuner 941 outputs an encoded bit stream obtainedby the demodulation to the selector 946. In other words, the tuner 941has a role of a transmission unit in the recording/reproducing device940.

The external interface 942 is the interface that connects between therecording/reproducing device 940 and an external appliance or a network.The external interface 942 may be, for example, the IEEE1394 interface,the network interface, the USB interface, or the flash memory interface.For example, the video data and audio data received through the externalinterface 942 are input to the encoder 943. In other words, the externalinterface 942 also has a role of a transmission unit in therecording/reproducing device 940.

If the video data and audio data input from the external interface 942have not been encoded, the encoder 943 encodes the video data and theaudio data. Then, the encoder 943 outputs the encoded bit stream to theselector 946.

The HDD 944 records the encoded bit stream containing compressed contentdata such as video and audio, various programs, and other pieces of datain the internal hard disk. The HDD 944 reads out these pieces of datafrom the hard disk when the video or audio is reproduced.

The disk drive 945 records and reads out the data into and from theattached recording medium. The recording medium attached to the diskdrive 945 may be, for example, a DVD disc (such as DVD-Video, DVD-RAM,DVD-R, DVD-RW, DVD+R, or DVD+RW) or a Blu-ray (registered trademark)disc.

When the video and audio are recorded, the selector 946 selects theencoded bit stream input from the tuner 941 or the encoder 943, andoutputs the selected encoded bit stream to the HDD 944 or the disk drive945. When the video and audio are reproduced, the selector 946 outputsthe encoded bit stream input from the HDD 944 or the disk drive 945 tothe decoder 947.

The decoder 947 decodes the encoded bit stream to generate the videodata and audio data. Then, the decoder 947 outputs the generated videodata to the OSD 948. The decoder 904 outputs the generated audio data tothe external speaker.

The OSD 948 reproduces the video data input from the decoder 947, anddisplays the video. The OSD 948 may overlap the GUI image such as amenu, a button, or a cursor on the displayed video.

The control unit 949 includes a processor such as a CPU, and a memorysuch as a RAM and a ROM. The memory stores programs to be executed bythe CPU, and program data, etc. The programs stored in the memory areread in and executed by the CPU when the recording/reproducing device940 is activated, for example. By executing the programs, the CPUcontrols the operation of the recording/reproducing device 940 inresponse to an operation signal input from the user interface 950, forexample.

The user interface 950 is connected to the control unit 949. The userinterface 950 includes, for example, a button and a switch for a user tooperate the recording/reproducing device 940, and a reception unit forreceiving a remote control signal. The user interface 950 generates theoperation signal by detecting the operation of the user through thesecomponents, and outputs the generated operation signal to the controlunit 949.

In the recording/reproducing device 940 with the above structure, theencoder 943 has a function of the image encoding device according to theabove embodiment. The decoder 947 has a function of the image decodingdevice according to the above embodiment. Thus, in the encoding anddecoding of the image in the recording/reproducing device 940, the blocknoise can be suppressed.

Fourth Application Example: Photographing Device

FIG. 26 illustrates an example of a schematic structure of aphotographing device to which the above embodiment has been applied. Aphotographing device 960 generates an image by photographing a subject,encodes the image data, and records the data in a recording medium.

The photographing device 960 includes an optical block 961, aphotographing unit 962, a signal processing unit 963, an imageprocessing unit 964, a display unit 965, an external interface 966, amemory unit 967, a media drive 968, an OSD 969, a control unit 970, auser interface 971, and a bus 972.

The optical block 961 is connected to the photographing unit 962. Thephotographing unit 962 is connected to the signal processing unit 963.The display unit 965 is connected to the image processing unit 964. Theuser interface 971 is connected to the control unit 970. The bus 972connects among the image processing unit 964, the external interface966, the memory 967, the media drive 968, the OSD 969, and the controlunit 970.

The optical block 961 has a focusing lens, a diaphragm mechanism, andthe like. The optical block 961 focuses an optical image of a subject ona photographing surface of the photographing unit 962. The photographingunit 962 includes an image sensor such as a CCD (Charge Coupled Device)or a CMOS (Complementary Metal Oxide Semiconductor), and converts theoptical image focused on the photographing surface into an image signalas an electric signal through photoelectric conversion. Then, thephotographing unit 962 outputs the image signal to the signal processingunit 963.

The signal processing unit 963 performs various camera signal processessuch as knee correction, gamma correction, and color correction on theimage signal input from the photographing unit 962. The signalprocessing unit 963 outputs the image data after the camera signalprocess, to the image processing unit 964.

The image processing unit 964 encodes the image data input from thesignal processing unit 963 and generates the encoded data. Then, theimage processing unit 964 outputs the generated encoded data to theexternal interface 966 or the media drive 968. The image processing unit964 decodes the encoded data input from the external interface 966 orthe media drive 968, and generates the image data. Then, the imageprocessing unit 964 outputs the generated image data to the display unit965. Moreover, the image processing unit 964 may output the image datainput from the signal processing unit 963 to the display unit 965 wherethe image is displayed. The image processing unit 964 may additionallyoverlap the display data acquired from the OSD 969 on the image outputto the display unit 965.

The OSD 969 generates the GUI image such as a menu, a button, or acursor and outputs the generated image to the image processing unit 964.

The external interface 966 is configured as, for example, a USBinput/output terminal. The external interface 966 connects, for example,between the photographing device 960 and a printer when the image isprinted. Moreover, the external interface 966 can have a drive connectedthereto when necessary. To the drive, for example, a removable mediumsuch as a magnetic disk or an optical disk is attached, and the programread out from the removable medium can be installed in the photographingdevice 960. Alternatively, the external interface 966 may be configuredas the network interface connected to the network such as LAN or theInternet. In other words, the external interface 966 has a role of atransmission unit in the photographing device 960.

The recording medium attached to the media drive 968 may be, forexample, any readable and writable removable medium such as a magneticdisk, a magneto-optic disk, an optical disk, or a semiconductor memory.The media drive 968 may have the recording medium fixedly attachedthereto and a non-transportable storage unit such as a built-in harddisk drive or an SSD (Solid State Drive) may be configured.

The control unit 970 includes a processor such as a CPU, and a memorysuch as a RAM and a ROM. The memory stores programs to be executed bythe CPU, and program data, etc. The programs stored in the memory areread in and executed by the CPU when the photographing device 960 isactivated, for example. By executing the programs, the CPU controls theoperation of the photographing device 960 in response to an operationsignal input from the user interface 971, for example.

The user interface 971 is connected to the control unit 970. The userinterface 971 includes, for example, a button and a switch for a user tooperate the photographing device 960. The user interface 971 generatesthe operation signal by detecting the operation of the user throughthese components, and outputs the generated operation signal to thecontrol unit 970.

In the photographing device 960 with the above structure, the imageprocessing unit 964 has a function of the image encoding device and theimage decoding device according to the above embodiment. Thus, in theencoding and decoding of the image in the photographing device 960, theblock noise can be suppressed.

7. Fifth Embodiment Another Example of Embodiment

The above description has been made of the examples of the device or thesystem to which the present technique has been applied; however, thepresent technique is not limited thereto. The present technique can becarried out as any kind of a structure mounted on the device as aboveand a structure included in the system, for example, to a processor as asystem LSI (Large Scale Integration), a module including a plurality ofprocessors, a unit including a plurality of modules, and a set havinganother function added to the unit (that is, the structure of a part ofthe device).

<Video Set>

An example of carrying out the present technique as a set is describedwith reference to FIG. 27. FIG. 27 illustrates an example of a schematicstructure of a video set to which the present technique has beenapplied.

In recent years, electronic appliances have come to have variousfunctions, and when just a part of the structure is sold or provided inthe development and manufacture, it is often seen that not just onestructure is provided but a plurality of structures with correlatedfunctions is combined and treated as one multi-functional set.

A video set 1300 illustrated in FIG. 27 has a structure with variousfunctions, which is formed by having a device with a function related toimage encoding or decoding (either one of them or both) combined with adevice with another function related to the above function.

As illustrated in FIG. 27, the video set 1300 includes a module groupincluding a video module 1311, an external memory 1312, a powermanagement module 1313, a front end module 1314, and the like, anddevices with the correlated functions, such as a connectivity 1321, acamera 1322, and a sensor 1323.

The module refers to a component with the functions obtained bycombining several correlated functions. The specific physical structureis arbitrarily given; for example, a plurality of electronic circuitelements each having its own function, such as a processor, a resistor,and a capacitor, and other devices may be disposed on a wiring board andintegrated. Further, another module or processor may be combined withthe above module to form a new module.

In the case of the example of FIG. 27, the video module 1311 is thecombination of the structures with functions related to imageprocessing, and includes an application processor, a video processor, abroadband modem 1333, and an RF module 1334.

The processor is formed by integrating structures with predeterminedfunctions on a semiconductor chip through SoC (System On a Chip), and isalso referred to as, for example, a system LSI (Large ScaleIntegration). The structure with the predetermined function may be alogic circuit (hardware structure), a CPU, a ROM, a RAM or the like, aprogram executed using the same (software structure), or the combinationof the both. For example, the processor may have a logic circuit, and aCPU, a ROM, a RAM, or the like and have a part of the function achievedby the logic circuit (hardware structure) and the other functions may beachieved by a program (software structure) executed in the CPU.

The application processor 1331 in FIG. 27 is the processor that executesthe application for the image processing. The application executed inthe application processor 1331 not just performs the calculation processbut also controls the structures in and out of the video module 1311such as the video processor 1332 as necessary for achieving thepredetermined function.

The video processor 1332 is the processor with the function for theimage encoding or decoding (one of them or both).

The broadband modem 1333 is a processor (or module) that performs theprocess related to the broadband communication with or (and) withoutwires through a broadband line such as the Internet or public telephonenetwork. For example, the broadband modem 1333 converts the transmissiondata (digital signals) into analog signals through digital modulation,or converts the received analog signal into the data (digital signals)by demodulating the received analog signal. The broadband modem 1333 candigitally modulate/demodulate any piece of information including theimage data, the stream in which the image data are encoded, theapplication program, and the setting data to be processed by the videoprocessor 1332, for example.

The RF module 1334 is the module that performs the frequency conversion,modulation/demodulation, amplification, filtering, and the like on theRF (Radio Frequency) signal that are exchanged via the antenna. Forexample, the RF module 1334 generates the RF signal by converting thefrequency of the base band signal generated by the broadband modem 1333.In another example, the RF module 1334 generates the base band signal byconverting the frequency of the RF signal received via the front endmodule 1314.

As indicated by a dotted line 1341 in FIG. 27, the application processor1331 and the video processor 1332 may be integrated into one processor.

The external memory 1312 is the module provided outside the video module1311 and having a storage device used by the video module 1311. Thestorage device of the external memory 1312 may be achieved by anyphysical structure; however, since the storage device is often used forstorage of a large amount of data such as the image data in units offrame, the storage device is desirably achieved by a semiconductormemory that has high capacity but costs less like a DRAM (Dynamic RandomAccess Memory).

The power management module 1313 manages and controls the power supplyto the video module 1311 (each structure in the video module 1311).

The front end module 1314 is the module that provides the RF module 1334with the front end function (circuit on the transmission/reception endof the antenna side). As illustrated in FIG. 27, the front end module1314 has, for example, an antenna unit 1351, a filter 1352, and anamplification unit 1353.

The antenna unit 1351 has an antenna that transmits and receiveswireless signals and a peripheral structure thereof. The antenna unit1351 transmits the signal supplied from the amplification unit 1353 andsupplies the received wireless signal to the filter 1352 as an electricsignal (RF signal). The filter 1352 filters the RF signal receivedthrough the antenna unit 1351 and supplies the processed RF signal tothe RF module 1334. The amplification unit 1353 amplifies the RF signalsupplied from the RF module 1334 and supplies the signal to the antennaunit 1351.

The connectivity 1321 is the module having the function related to theexternal connection. The physical structure of the connectivity 1321 maybe determined arbitrarily. For example, the connectivity 1321 has astructure with a communication function other than the communicationspecification for the broadband modem 1333, an external input/outputterminal, or the like.

For example, the connectivity 1321 may have a module with acommunication function based on the wireless communication specificationsuch as Bluetooth (registered trademark), IEEE 802.11 (for example,Wi-Fi (Wireless Fidelity, registered trademark)), NFC (Near FieldCommunication), or IrDA (InfraRed Data Association), an antenna thattransmits or receives the signal based on those specifications, or thelike. Alternatively, the connectivity 1321 may have a module with acommunication function based on the wired communication specificationsuch as USB (Universal Serial Bus) or HDMI (registered trademark,High-Definition Multimedia Interface), a terminal based on thatspecification, or the like. Further alternatively, the connectivity 1321may have another data (signal) transmission function such as an analoginput/output terminal, or the like.

Note that the connectivity 1321 may incorporate a device to which thedata (signals) are transmitted. For example, the connectivity 1321 mayhave a drive (not just the drive of the removable medium but also a harddisk, an SSD (Solid State Drive), or a NAS (Network Attached Storage))that reads out or writes data from and in a recording medium such as amagnetic disk, an optical disk, a magneto-optical disk, or asemiconductor memory. The connectivity 1321 may have a device thatoutputs the image or audio (monitor, speaker, or the like).

The camera 1322 is the module that photographs a subject and obtainsimage data of the subject. The image data obtained by the photographingwith the camera 1322 are supplied to, for example, the video processor1332 and encoded therein.

The sensor 1323 is the module with any sensor function, such as an audiosensor, an ultrasonic wave sensor, a light sensor, an illuminancesensor, an infrared-ray sensor, an image sensor, a rotation sensor, anangle sensor, an angular velocity sensor, a speed sensor, anacceleration sensor, an inclination sensor, a magnetic identificationsensor, a shock sensor, or a temperature sensor. The data detected bythe sensor 1323 are supplied to the application processor 1331 and usedby the application, etc.

The structure described above as the module may be achieved as aprocessor or on the contrary, the structure described as the processormay be achieved as a module.

In the video set 1300 with the above structure, the present techniquecan be applied to the video processor 1332 as described below.Therefore, the video set 1300 can be used as the set to which thepresent technique has been applied.

[Structure Example of Video Processor]

FIG. 28 illustrates an example of a schematic structure of the videoprocessor 1332 (FIG. 27) to which the present technique has beenapplied.

In the case of the example of FIG. 28, the video processor 1332 has afunction of encoding a video signal and an audio signal by apredetermined method upon the reception of the signals, and a functionof decoding the encoded video data and audio data and reproducing andoutputting the video signal and the audio signal.

As illustrated in FIG. 28, the video processor 1332 includes a videoinput processing unit 1401, a first image magnifying/reducing unit 1402,a second image magnifying/reducing unit 1403, a video output processingunit 1404, a frame memory 1405, and a memory control unit 1406. Thevideo processor 1332 includes an encode/decode engine 1407, video ES(Elementary Stream) buffers 1408A and 1408B, and audio ES buffers 1409Aand 1409B. The video processor 1332 further includes an audio encoder1410, an audio decoder 1411, a multiplexing unit (MUX (Multiplexer))1412, a demultiplexing unit (DMUX (Demultiplexer)) 1413, and a streambuffer 1414.

The video input processing unit 1401 acquires the video signal inputfrom, for example, the connectivity 1321 (FIG. 27) and converts thesignal into digital image data. The first image magnifying/reducing unit1402 performs the format conversion on the image data ormagnifies/reduces the image. The second image magnifying/reducing unit1403 performs the image magnifying/reducing process on the image dataaccording to the format at the destination to which the data are outputthrough the video output processing unit 1404, or performs the formatconversion on the image data or magnifies/reduces the image in a mannersimilar to the first image magnifying/reducing unit 1402. The videooutput processing unit 1404 performs the format conversion on the imagedata or converts the image data into analog signals, for example, andoutputs the data as the reproduced video signal to the connectivity 1321(FIG. 27), for example.

The frame memory 1405 is the memory for the image data shared by thevideo input processing unit 1401, the first image magnifying/reducingunit 1402, the second image magnifying/reducing unit 1403, the videooutput processing unit 1404, and the encode/decode engine 1407. Theframe memory 1405 is achieved as, for example, a semiconductor memorysuch as a DRAM.

The memory control unit 1406 controls the access of writing and readingin and from the frame memory 1405 according to the access schedule forthe frame memory 1405 that has been written in an access managementtable 1406A upon the reception of a synchronization signal from theencode/decode engine 1407. The access management table 1406A is updatedby the memory control unit 1406 in response to the process executed bythe encode/decode engine 1407, the first image magnifying/reducing unit1402, the second image magnifying/reducing unit 1403, or the like.

The encode/decode engine 1407 encodes the image data and decodes thevideo stream, which is the data obtained by encoding the image data. Forexample, the encode/decode engine 1407 encodes the image data read outfrom the frame memory 1405, and sequentially writes the data in thevideo ES buffer 1408A as the video streams. Moreover, the encode/decodeengine 1407 sequentially reads out the video streams from the video ESbuffer 1408B and decodes the streams, and sequentially writes thestreams in the frame memory 1405 as the image data. The encode/decodeengine 1407 uses the frame memory 1405 as a work region in the encodingand decoding. The encode/decode engine 1407 outputs a synchronizationsignal to the memory control unit 1406 at the timing at which theprocess for each macroblock is started, for example.

The video ES buffer 1408A buffers the video stream generated by theencode/decode engine 1407, and supplies the stream to the multiplexingunit (MUX) 1412. The video ES buffer 1408B buffers the video streamsupplied from the demultiplexing unit (DMUX) 1413 and supplies thestream to the encode/decode engine 1407.

The audio ES buffer 1409A buffers the audio stream generated by theaudio encoder 1410, and supplies the stream to the multiplexing unit(MUX) 1412. The audio ES buffer 1409B buffers the audio stream suppliedfrom the demultiplexing unit (DMUX) 1413 and supplies the stream to theaudio decoder 1411.

The audio encoder 1410 converts the audio signal input from theconnectivity 1321 (FIG. 27), for example, into the digital signals,thereby encoding the signal by a predetermined method such as the MPEGaudio method or AC3 (AudioCode number 3). The audio encoder 1410sequentially writes the audio stream as the data obtained by encodingthe audio signals into the audio ES buffer 1409A. The audio decoder 1411decodes the audio stream supplied from the audio ES buffer 1409B, andconverts the stream into analog signals, and then supplies the signalsas the reproduced audio signals to, for example, the connectivity 1321(FIG. 27).

The multiplexing unit (MUX) 1412 multiplexes the video stream and theaudio stream. A method for this multiplexing (i.e., format of the bitstream generated by the multiplexing) may be determined arbitrarily. Inthe multiplexing, the multiplexing unit (MUX) 1412 may add predeterminedheader information or the like to the bit stream. In other words, themultiplexing unit (MUX) 1412 can convert the format of the stream by themultiplexing. For example, the multiplexing unit (MUX) 1912 multiplexesthe video stream and the audio stream to convert the streams into thetransport stream, which is the bit stream in the format for transfer. Inanother example, the multiplexing unit (MUX) 1412 multiplexes the videostream and the audio stream to convert the streams into the data (filedata) in the file format for recording.

The demultiplexing unit (DMUX) 1413 demultiplexes the bit stream inwhich the video stream and the audio stream are multiplexed, by a methodcorresponding to the multiplexing by the multiplexing unit (MUX) 1412.In other words, the demultiplexing unit (DMUX) 1413 extracts the videostream and the audio stream from the bit streams read out of the streambuffer 1414 (separates the video stream and the audio stream from eachother). That is to say, the demultiplexing unit (DMUX) 1413 can convertthe format of the stream by demultiplexing (inverted conversion of theconversion by the multiplexing unit (MUX) 1412). For example, thedemultiplexing unit (DMUX) 1413 acquires the transport stream suppliedfrom the connectivity 1321 or the broadband modem 1333 (both illustratedin FIG. 27) through the stream buffer 1414, and demultiplexes thestream, whereby the transport stream can be converted into the videostream and the audio stream. In another example, the demultiplexing unit(DMUX) 1413 acquires the file data read out from any of variousrecording media by the connectivity 1321 (FIG. 27) through the streambuffer 1414, and demultiplexes the stream, whereby the data can beconverted into the video stream and the audio stream.

The stream buffer 1414 buffers the bit stream. For example, the streambuffer 1414 buffers the transport stream supplied from the multiplexingunit (MUX) 1412, and supplies the stream to the connectivity 1321 or thebroadband modem 1333 (both illustrated in FIG. 27) at a predeterminedtiming or upon a request from the outside.

In another example, the stream buffer 1414 buffers the file datasupplied from the multiplexing unit (MUX) 1412 and supplies the data tothe connectivity 1321 (FIG. 27) or the like at a predetermined timing orupon a request from the outside, and records the data in any of variousrecording media.

Further, the stream buffer 1414 buffers the transport stream acquiredthrough the connectivity 1321 or the broadband modem 1333 (bothillustrated in FIG. 27), and supplies the stream to the demultiplexingunit (DMUX) 1413 at a predetermined timing or upon a request from theoutside.

The stream buffer 1414 buffers the file data read out from any ofvarious recording media at the connectivity 1321 (FIG. 27) or the like,and supplies the data to the demultiplexing unit (DMUX) 1413 at apredetermined timing or upon a request from the outside.

Next, an example of the operation of the video processor 1332 with theabove structure is described. For example, the video signal input to thevideo processor 1332 from the connectivity 1321 (FIG. 27) or the like isconverted into the digital image data of a predetermined format such as4:2:2Y/Cb/Cr format in the video input processing unit 1401, and theimage data are sequentially written in the frame memory 1405. Thedigital image data are read out by the first image magnifying/reducingunit 1402 or the second image magnifying/reducing unit 1403, aresubjected to the format conversion into a predetermined format such as4:2:0Y/Cb/Cr format and magnified or reduced, and then written in theframe memory 1405 again. The image data are encoded by the encode/decodeengine 1407 and written in the video ES buffer 1408A as the videostream.

Moreover, the audio signal input from the connectivity 1321 (FIG. 27) orthe like to the video processor 1332 is encoded by the audio encoder1410, and written in the audio ES buffer 1409A as the audio stream.

The video stream of the video ES buffer 1408A and the audio stream ofthe audio ES buffer 1409A are read out by the multiplexing unit (MUX)1412 and multiplexed therein to be converted into the transport streamor the file data, for example. The transport stream generated by themultiplexing unit (MUX) 1412 is buffered by the stream buffer 1414, andoutput to the external network through the connectivity 1321 or thebroadband modem 1333 (both illustrated in FIG. 27). The file datagenerated by the multiplexing unit (MUX) 1412 are buffered by the streambuffer 1414 and then output to the connectivity 1321 (FIG. 27) or thelike, and then recorded in any of various recoding media.

The transport stream input to the video processor 1332 from the externalnetwork through the connectivity 1321 or the broadband modem 1333 (bothillustrated in FIG. 27) is buffered by the stream buffer 1414 and thendemultiplexed by the demultiplexing unit (DMUX) 1413. The file data readout from any of various recording media and input to the video processor1332 at the connectivity 1321 (FIG. 27) or the like are buffered by thestream buffer 1414 and then demultiplexed by the demultiplexing unit(DMUX) 1413. In other words, the file data or the transport stream inputto the video processor 1332 is separated into the video stream and theaudio stream by the demultiplexing unit (DMUX) 1413.

The audio stream is supplied to the audio decoder 1411 through the audioES buffer 1409B and decoded, so that the audio signal is reproduced. Thevideo stream is written in the video ES buffer 1408B and sequentiallyread out by the encode/decode engine 1407 and decoded, and written inthe frame memory 1405. The decoded image data are magnified or reducedby the second image magnifying/reducing unit 1403 and written in theframe memory 1405. The decoded image data are read out by the videooutput processing unit 1404 and subjected to the format conversion intoa predetermined format such as 4:2:2Y/Cb/Cr format, and are furtherconverted into an analog signal, so that the video signal is reproducedand output.

When the present technique is applied to the video processor 1332 withthis structure, the present technique according to any of the aboveembodiments may be applied to the encode/decode engine 1407. Forexample, the encode/decode engine 1407 may have a function of the imageencoding device 11 (FIG. 1) and the image decoding device 51 (FIG. 3)according to the first embodiment. Thus, the video processor 1332 canprovide the effect similar to the above effect described with referenceto FIG. 1 to FIG. 15.

In the encode/decode engine 1407, the present technique (i.e., thefunction of the image encoding device and the image decoding deviceaccording to any of the above embodiments) may be achieved by hardwaresuch as a logic circuit or software such as a built-in program, or boththe hardware and the software.

[Another Structure Example of Video Processor]

FIG. 29 illustrates another example of a schematic structure of thevideo processor 1332 (FIG. 27) to which the present technique has beenapplied. In the case of the example of FIG. 29, the video processor 1332has a function of encoding and decoding video data in a predeterminedformat.

More specifically, as illustrated in FIG. 29, the video processor 1332includes a control unit 1511, a display interface 1512, a display engine1513, an image processing engine 1514, and an internal memory 1515. Thevideo processor 1332 includes a codec engine 1516, a memory interface1517, a multiplexing/demultiplexing unit (MUX/DMUX) 1518, a networkinterface 1519, and a video interface 1520.

The control unit 1511 controls the operation of the processing units inthe video processor 1332, such as the display interface 1512, thedisplay engine 1513, the image processing engine 1514, and the codecengine 1516.

As illustrated in FIG. 29, the control unit 1511 includes, for example,a main CPU 1531, a sub CPU 1532, and a system controller 1533. The mainCPU 1531 executes the programs and the like for controlling theoperation of the processing units in the video processor 1332. The mainCPU 1531 generates control signals in accordance with the programs andthe like and supplies the signals to the processing units (i.e.,controls the operation of the processing units). The sub CPU 1532 servesto assist the main CPU 1531. For example, the sub CPU 1532 executes thechild process or subroutine of the programs executed by the main CPU1531. The system controller 1533 controls the operation of the main CPU1531 and the sub CPU 1532; for example, the system controller 1533specifies the programs to be executed by the main CPU 1531 and the subCPU 1532.

The display interface 1512 outputs the image data to, for example, theconnectivity 1321 (FIG. 27) under the control of the control unit 1511.For example, the display interface 1512 converts the digital image datato the analog signals and outputs the data as the reproduced videosignal or image data in the form of digital data to the monitor deviceor the like of the connectivity 1321 (FIG. 27).

Under the control of the control unit 1511, the display engine 1513performs various conversion processes such as format conversion, sizeconversion, and color range conversion on the image data to suit thespecification of the hardware such as the monitor device where the imageis displayed.

The image processing engine 1514 performs a predetermined image processsuch as filtering for image quality improvement on the image data underthe control of the control unit 1511.

The internal memory 1515 is the memory provided in the video processor1332 and is shared among the display engine 1513, the image processingengine 1514, and the codec engine 1516. The internal memory 1515 is usedto exchange data among the display engine 1513, the image processingengine 1514, and the codec engine 1516. For example, the internal memory1515 stores the data supplied from the display engine 1513, the imageprocessing engine 1514, and the codec engine 1516, and supplies the datato the display engine 1513, the image processing engine 1514, or thecodec engine 1516 as necessary (or upon a request). The internal memory1515 may be formed by any kind of storage device; generally, since thememory is used to store a small amount of data such as the image data orparameters in units of block, the memory 1515 is desirably formed by asemiconductor memory that has relatively small capacity (smallercapacity than the external memory 1312) but has high response speed,such as an SRAM (Static Random Access Memory).

The codec engine 1516 performs the processes for encoding or decodingthe image data. The method of encoding or decoding by the codec engine1516 is determined arbitrarily and the number of methods may be one ormore than one. For example, the codec engine 1516 may have a codecfunction of a plurality of encoding/decoding methods, and may encode theimage data or decode the encoded data by the selected method.

In the example illustrated in FIG. 29, the codec engine 1516 has, forexample, MPEG-2 Video 1541, AVC/H.264 1542, HEVC/H.265 1543, HEVC/H.265(Scalable) 1544, HEVC/H.265 (Multi-view) 1545, and MPEG-DASH 1551 as thefunction blocks of the process related to the codec.

MPEG-2 Video 1541 corresponds to the function block that encodes ordecodes the image data in the MPEG-2 method. AVC/H.264 1542 correspondsto the function block that encodes or decodes the image data in the AVCmethod. HEVC/H.265 1543 corresponds to the function block that encodesor decodes the image data in the HEVC method. HEVC/H.265 (Scalable) 1544corresponds to the function block that scalably encodes or scalablydecodes the image data in the HEVC method. HEVC/H.265 (Multi-view) 1545corresponds to the function block that encodes or decodes the image datawith multiple viewpoints in the HEVC method.

MPEG-DASH 1551 corresponds to the function block that transmits orreceives the image data in the MPEG-DASH (MPEG-Dynamic AdaptiveStreaming over HTTP) method. MPEG-DASH is the technique of streaming thevideo using HTTP (HyperText Transfer Protocol), and one feature thereofis to select and transmit the appropriate piece of prepared encoded datawith different resolutions, etc. in units of segment. MPEG-DASH 1551generates the stream based on the specification or controls thetransmission of the stream, and uses the aforementioned MPEG-2 Video1541 to HEVC/H.265 (Multi-view) 1545 in the encoding and decoding of theimage data.

The memory interface 1517 is the interface for the external memory 1312.The data supplied from the image processing engine 1514 or the codecengine 1516 are supplied to the external memory 1312 through the memoryinterface 1517. The data read out from the external memory 1312 aresupplied to the video processor 1332 (image processing engine 1514 orcodec engine 1516) through the memory interface 1517.

The multiplexing/demultiplexing unit (MUX/DMUX) 1518 multiplexes ordemultiplexes various pieces of data related to the image, such as thebit stream of the encoded data, the image data, and the video signals.The method of the multiplexing/demultiplexing is determined arbitrarily.For example, in the multiplexing, in addition to collecting the pluralpieces of data, the multiplexing/demultiplexing unit (MUX/DMUX) 1518 canadd predetermined header information, etc. to the collected data. On thecontrary, in the demultiplexing, in addition to dividing the data intoplural pieces, the multiplexing/demultiplexing unit (MUX/DMUX) 1518 canadd predetermined header information to the divided piece of data. Inother words, the multiplexing/demultiplexing unit (MUX/DMUX) 1518 canconvert the format of the data by the multiplexing/demultiplexing. Forexample, the multiplexing/demultiplexing unit (MUX/DMUX) 1518 canconvert the bit stream into the transport stream, which is the bitstream in the format for transport, or the data in the file format forrecording (file data) by multiplexing the bit stream. Needless to say,the inverse conversion is also possible by the demultiplexing.

The network interface 1519 is the interface for the broadband modem 1333and the connectivity 1321 (both illustrated in FIG. 27), etc. The videointerface 1520 is the interface for the connectivity 1321 and the camera1322 (both illustrated in FIG. 27), etc.

Next, an example of the operation of the video processor 1332 as aboveis described. For example, upon the reception of the transport streamfrom the external network through the connectivity 1321 or the broadbandmodem 1333 (both illustrated in FIG. 27), the transport stream issupplied to the multiplexing/demultiplexing unit (MUX/DMUX) 1518 throughthe network interface 1519, demultiplexed therein, and decoded by thecodec engine 1516. The image data obtained by the decoding of the codecengine 1516 are subjected to a predetermined image process by the imageprocessing engine 1514, for example, and to a predetermined conversionby the display engine 1513 and the data are supplied to the connectivity1321 (FIG. 27) or the like through the display interface 1512; thus, theimage is displayed on the monitor. In another example, the image dataobtained by the decoding of the codec engine 1516 are encoded again bythe codec engine 1516 and multiplexed by the multiplexing/demultiplexingunit (MUX/DMUX) 1518 and converted into the file data; then, the dataare output to the connectivity 1321 (FIG. 27) or the like through thevideo interface 1520 and recorded in any of various recording media.

In another example, the file data of the encoded data in which the imagedata are encoded, which have been read out from the unshown recordingmedium, by the connectivity 1321 (FIG. 27) or the like are supplied tothe multiplexing/demultiplexing unit (MUX/DMUX) 1518 through the videointerface 1520 and demultiplexed therein, and decoded by the codecengine 1516. The image data obtained by the decoding of the codec engine1516 are subjected to a predetermined image process by the imageprocessing engine 1514, and to a predetermined conversion by the displayengine 1513; then, the data are supplied to the connectivity 1321 (FIG.27) or the like through the display interface 1512 and the image isdisplayed on the monitor. In another example, the image data obtained bythe decoding of the codec engine 1516 are encoded again by the codecengine 1516 and multiplexed by the multiplexing/demultiplexing unit(MUX/DMUX) 1518 and converted into the transport stream; then, the dataare supplied to the connectivity 1321 or the broadband modem 1333 (bothillustrated in FIG. 27) or the like through the network interface 1519and transmitted to another device which is not shown.

The image data or another piece of data is exchanged among theprocessing units in the video processor 1332 through, for example, theinternal memory 1515 or the external memory 1312. The power managementmodule 1313 controls the power supply to the control unit 1511, forexample.

In the case of applying the present technique to the video processor1332 with the structure as above, the present technique according to anyof the above embodiments may be applied to the codec engine 1516. Inother words, for example, the codec engine 1516 may have the functionblock that achieves the image encoding device 11 (FIG. 1) and the imagedecoding device 51 (FIG. 3) according to the first embodiment. Thus, thevideo processor 1332 can provide the effect similar to the above effectdescribed with reference to FIG. 1 to FIG. 15.

In the codec engine 1516, the present technique (i.e., the function ofthe image encoding device and the image decoding device according to anyof the above embodiments) may be achieved by hardware such as a logiccircuit or software such as a built-in program, or both the hardware andthe software.

The two examples have been described as the structure of the videoprocessor 1332; however, the structure of the video processor 1332 maybe determined arbitrarily and may be other than the above two examples.The video processor 1332 may be configured as one semiconductor chip oras a plurality of semiconductor chips. For example, a three-dimensionalmultilayer LSI in which a plurality of semiconductor layers are stackedmay be used. Alternatively, a plurality of LSIs may be used.

[Example of Application to Device]

The video set 1300 can be incorporated into various devices that processthe image data. For example, the video set 1300 can be incorporated inthe television device 900 (FIG. 23), the cellular phone 920 (FIG. 24),the recording/reproducing device 940 (FIG. 25), the photographing device960 (FIG. 26), or the like. By having the video set 1300 incorporated,the device can have the effect similar to the effect described withreference to FIG. 1 to FIG. 15.

Even a part of each structure of the above video set 1300 can beregarded as the structure to which the present technique has beenapplied as long as the structure includes the video processor 1332. Forexample, just the video processor 1332 can be embodied as the videoprocessor to which the present technique has been applied. Further, forexample, as described above, the processor or the video module 1311,which is illustrated by the dotted line 1341, can be embodied as theprocessor or the module to which the present technique has been applied.Moreover, for example, the video module 1311, the external memory 1312,the power management module 1313, and the front end module 1314 can becombined to be embodied as a video unit 1361 to which the presenttechnique has been applied. In any structure, the effect similar to theeffect described with reference to FIG. 1 to FIG. 15 can be obtained.

As long as the structure includes the video processor 1332, thestructure can be incorporated in various devices that process image datain a manner similar to the video set 1300. For example, the videoprocessor 1332, the processor indicated by the dotted line 1341, thevideo module 1311, or the video unit 1361 can be incorporated in thetelevision device 900 (FIG. 23), the cellular phone 920 (FIG. 24), therecording/reproducing device 940 (FIG. 25), or the photographing device960 (FIG. 26). By incorporating any of the above structures to which thepresent technique has been applied, the device can have the effectsimilar to the effect described with reference to FIG. 1 to FIG. 15, ina manner similar to the case of the video set 1300.

In this specification, description has been made of the example in whichvarious pieces of information such as the parameter of the deblockingfilter and the parameter of the adaptive offset filter are multiplexedon the encoded stream and transmitted from the encoding side to thedecoding side. The method of transmitting the information, however, isnot limited to this example. For example, these pieces of informationmay be transmitted or recorded as the separate pieces of data that arecorrelated to the encoded bit stream without being multiplexed on theencoded bit stream. Here, “correlation” refers to the link of the imageincluded in the bit stream (may be a part of the image such as slice orblock) and the information corresponding to the image at the decoding.In other words, the information may be transmitted on a transmissionpath separate from the image (or bit stream). Alternatively, theinformation may be recorded in a recording medium separate from theimage (or bit stream) (or in another recording area of the samerecording medium). The information and the image (or bit stream) may becorrelated to each other in any unit, such as in a plurality of frames,one frame, or a part of a frame.

The preferred embodiments of the present disclosure have been describedwith reference to the attached drawings; however, the present disclosureis not limited to the examples above. It is apparent that a personskilled in the art to which the present disclosure pertains can conceivevarious modifications or improvements in the scope of technical thoughtsdescribed in the scope of claims, and those are included in the range ofthe technique according to the present disclosure.

The present technique can have any of the structures described below.

(1) An image processing device including:

a filter determination unit that determines whether to apply a strongfilter by using a strength parameter representing strength of adeblocking filter for a block boundary, which is a boundary between ablock of an image locally decoded when an image is encoded, and anadjacent block adjacent to the block;

a filter unit that applies the strong filter for the block boundary ifthe filter determination unit has determined to apply the strong filter;and

an encoding unit that encodes the image in units having a layerstructure by using the image to which the strong filter has been appliedby the filter unit.

(2) The image processing device according to (1), wherein the filterdetermination unit determines whether to apply the strong filter byusing the strength parameter representing the strength of the deblockingfilter for the block boundary and a determination parameter representinga threshold used in the determination of the deblocking filter.

(3) The image processing device according to (2), wherein whether toapply the strong filter is determined using a value obtained bymultiplying the strength parameter by the determination parameter.

(4) The image processing device according to (2), wherein the filterdetermination unit determines whether to apply the strong filter byusing a function in which the strength parameter or the determinationparameter is used as a variable.

(5) The image processing device according to any of (2) to (4), whereinthe filter determination unit determines whether to apply the strongfilter by using a linear function of the strength parameter and thedetermination parameter.

(6) The image processing device according to any of (2) to (5), whereinthe strength parameter is a parameter that is larger when a predictionmode is intra prediction than when the prediction mode is interprediction.

(7) The image processing device according to any of (2) to (6), whereinthe strength parameter is Bs (Boundary Filtering Strength) value.

(8) The image processing device according to any of (2) to (7), whereinthe determination parameter is beta used in the determination of whetherto apply the deblocking filter and the determination of strengthselection of the deblocking filter.

(9) An image processing method wherein an image processing deviceperforms the steps of:

determining whether to apply a strong filter by using a strengthparameter representing strength of a deblocking filter for a blockboundary, which is a boundary between a block of an image locallydecoded when an image is encoded, and an adjacent block adjacent to theblock;

applying the strong filter for the block boundary if it has beendetermined to apply the strong filter; and

encoding the image in units having a layer structure by using the imageto which the strong filter has been applied.

REFERENCE SIGNS LIST

-   11 Image encoding device-   31, 31 a, 31 b Deblocking filter-   51 Image decoding device-   101 Image memory-   102 Block boundary determination unit-   103 Filter strength determination unit-   104 Filter calculation unit-   105 Coefficient memory-   106 Control unit

The invention claimed is:
 1. An image processing device comprising:circuitry configured to determine, for a first layer of an image havinga plurality of layers, whether to apply a deblocking filter including afirst deblocking filter and a second deblocking filter having a strongerfilter than the first deblocking filter by using a strength parameterrepresenting strength of a deblocking filter for a block boundary, whichis a boundary between a block of the image locally decoded when an imageis encoded, and an adjacent block adjacent to the block; apply thesecond deblocking filter for the block boundary if the circuitry hasdetermined to apply the second deblocking filter; apply, to anotherlayer of the image, at least one of the strength parameter and a resultof determination of whether to apply the deblocking filter; and encodethe image in coding blocks formed by recursively splitting a coding treeblock into smaller coding blocks as block partitioning by using theimage to which the second deblocking filter has been applied, whereinthe circuitry is configured to determine whether to apply the seconddeblocking filter by using the strength parameter representing thestrength of the deblocking filter for the block boundary and adetermination parameter representing a threshold used in thedetermination of the deblocking filter, and wherein whether to apply thesecond deblocking filter is determined by obtaining the determinationfilter from a table and then comparing a value obtained by multiplyingthe strength parameter by the determination parameter with a threshold.2. The image processing device according to claim 1, wherein thecircuitry is configured to determine whether to apply the seconddeblocking filter by using a function in which the strength parameter orthe determination parameter is used as a variable.
 3. The imageprocessing device according to claim 2, wherein the circuitry isconfigured to determine whether to apply the second deblocking filter byusing a linear function of the strength parameter and the determinationparameter.
 4. The image processing device according to claim 1, whereinthe strength parameter is a parameter that is larger when a predictionmode is intra prediction than when the prediction mode is interprediction.
 5. The image processing device according to claim 4, whereinthe strength parameter is Bs (Boundary Filtering Strength) value.
 6. Theimage processing device according to claim 1, wherein the determinationparameter is beta used in the determination of whether to apply thedeblocking filter and the determination of strength selection of thedeblocking filter.
 7. An image processing method wherein an imageprocessing device performs the steps of: determining, using circuitry,for a first layer of an image having a plurality of layers, whether toapply a deblocking filter including a first deblocking filter and asecond deblocking filter having a stronger filter than the firstdeblocking filter by using a strength parameter representing strength ofa deblocking filter for a block boundary, which is a boundary between ablock of an image locally decoded when an image is encoded, and anadjacent block adjacent to the block; applying, using the circuitry, thesecond deblocking filter for the block boundary if it has beendetermined to apply the second deblocking filter; applying, using thecircuitry, to another layer of the image, at least one of the strengthparameter and a result of determination of whether to apply thedeblocking filter; and encoding, using the circuitry, the image incoding blocks formed by recursively splitting a coding tree block intosmaller coding blocks as block partitioning by using the image to whichthe second deblocking filter has been applied, wherein whether to applythe second deblocking filter is determined by using the strengthparameter representing the strength of the deblocking filter for theblock boundary and a determination parameter representing a thresholdused in the determination of the deblocking filter, and wherein whetherto apply the second deblocking filter is determined by obtaining thedetermination filter from a table and then comparing a value obtained bymultiplying the strength parameter by the determination parameter with athreshold.